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ABSTRACT 


The  search  for  suid  development  of  algorithms  for  a  real-time,  auto¬ 
matic  system  capable  of  tracking  2-D  targets  in  complex  scenes  is  dis¬ 
cussed.  The  problems  and  constraints  involved  in  such  an  undertcUcing 
as  well  as  previous  efforts  cure  summarized.  No  algorithms  could  be  found 
in  the  literature  that  will  provide  satisfactory  results  for  realistic 
conditions .  A  mathematical  model  of  scene  spatial  euid  temporal  evolu¬ 
tion  is  developed  for  certain  classes  of  targets  and  taxget  perturbations. 
It  is  shown  that  for  small  target  perturbations  the  2-D  tracking  prob¬ 
lem  may  be  approximated  as  a  1-D  time-varying  parameter  estimation  prob¬ 
lem.  A  novel  auid  CCD-implementable  solution  is  developed  which  is  super¬ 
ior  to  previously  developed  algorithms ,  particularly  when  considering 
target  rotation  and  dilation  and  target/background  separation.  Results 
of  system  simulation  amd  suggestions  for  future  research  are  presented. 


■  CHAPTER  Z 


ZHTROroCTZOn 

1.1  Baeicqromd 

As  the  sp««d,  capability  and  aconanlc  advantages  of  aodean  signal 
processing  devices  continue  to  increase,  there  is  simultaneously  an  in¬ 
crease  in  efforts  aimed  at  developing  sophisticated,  real-tim  autcnatic 
systems  capable  of  emulating  htanan  (or  perhaps  more  generally,  biologi¬ 
cal)  functicns.  the  research  described  herein  concerns  the  developisent 
and  implementation  of  algorithms  for  a  real-time  video  tracking  system, 
in  particular,  a  dedicated  discrete-analog  coaputer  capable  of  process¬ 
ing  complex,  time-varying  visual  information  in  order  to  extract  the 
camera  mount  control  or  guidance  signals  necessary  to  follow  a  predeter¬ 
mined  visual  target. 

Ae  attespted  autcsMtian  of  this  c^>ability  results  in  an  ex¬ 
tremely  ccoplicated  problem,  this  is  at  least  partially  due  to  the  fact 
that  "...the  perceptual  processes  involved  in... the  human  visxial  system 
...are  not  well  understood..."  [1],  and  perhaps  accounts  for  the  many 

unsatisfactory  empirical  and  heuristic  solutions  which  have  been  pro- 

* 

posed. 

While  a  more  exact  mathematical  problem  description  will  be  un¬ 
dertaken  later,  basically  (see  Figure  1)  the  problem  addressed  herein 
is  as  follows:  The  target  to  be  tracked  is  located  through  a  viewing 
system  (a  CCD  camera  and  motnt  with  asimuth,  elevation  and  zoom  con¬ 
trols  is  assumed)  in  such  a  way  that  automatic  feature  extraction  is 
made  possible.  Once  the  target  is  located,  it  is  desired  that  the 
automatic  system  provide  control  signals  to  the  camera  mount  to  keep 
the  target  centered  in  the  camera  field  of  view  (FOV)  ,  in  spite  of 
target  translation,  rotation  (3-0),  magnification,  and  possibly  tem¬ 
porary  occlusion  (partial  or  total).  Thus,  the  overall  problem  is  to 
design  a  "2-0  regulator"  for  a  closed  loop  control  system  with  both 


1 


2 


1-0  and  2-D  "Uaka,"^  and  thazafora  it  sbonld  ba  ajqpactad  that  both  con¬ 
trol  and  inaqa  pxocasslng  thaory  will  ba  aaployad  in  tha  solution.  Tbm 
military  and  industrial  avpiications'  for  such  a  sophisticatad  systaa 
paar  prcoising,  particularly  if  tha  antira  OCD  procassor  could  ba  fabri- 
catad  on  a  singla  circuit  board.  For  axas^la,  high-laval  automatic  air¬ 
craft  tracJcars  or  assambl  y  lina  conputar  vision  systams  could  ba  ij^>la- 
mantad. 


1.2  Haport  Organisation  and  s»"™»j^ry 
This  zaport  suanarizas  resaarch  which  lad  to  tha  development  of  a 
sat  of  naw,  unigua,  and  CCD  Isplemantabla  video  processing  algorithms 
cooprising  vrtiat  is  tazaad  tha  Taylor  Series  Video  Zmaga  Procassor 
(TSVIP) .  Basically,  this  approach  aliainatas  many  of  tha  severe  short¬ 
comings  of  previous  efforts,  including  tha  assumption  of  u«ni«n«ri 
simple  targets  and  scenes,  unsuitability  for  real-time  operation,  and 
inability  to  handle  perspective  transformations. 

Tha  report  is  organized  as  follows:  Oiaptar  II  presents  a  more 
view  of  tha  systaa  design  concept,  necessary  algorithas  for  a 
successful  system,  desirable  systaa  performance  features,  limitations, 
and  conpariscn  of  tha  desired  system  with  tha  human  visual  systaa  (HVS) ; 
Chapter  III  reviews  previous  work,  including  shortcomings;  Chapter  TV 
discusses  tha  mathematical  details  of  the  problaa  and  develops  a  model 
for  scene  spatial  and  temporal  evolution;  and  Chapter  V  presents  a 
unique  solutlco,  termed  the  Taylor  Series  Video  Image  Processor  (TSVIP). 
Qjspter  VI  summarizes  conputar  siaulatioo  undertaken  to  confirm  the  va¬ 
lidity  of  the  proposed  theory.  Finally,  Chapter  VII  susmarizes  the  work 


1)  A  note  here  should  be  made  concerning  tha  term  "dimension."  Unless 
otherwise  stated,  this  term  refers  to  the  number  of  arguments  of  a  (con¬ 
tinuous  or  discrete)  function,  not  the  size  of  a  vector.  For  exaaple, 
using  this  convention,  P(x,t)  is  a  2-diasnaional  function,  whereas  x(t) 

•  one-diaensional  (vector)  function  (of  t) ,  even  though  may  be 'a 
nxl  vector,  vrtiich,  in  referring  to  its  size,  is  commonly  termed  an  "N 
dimensional  vector." 


Figure  1  Video  Tracking  Syotem  Example 
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gtS'JZM  OVERVZJEtf 

D<«lrabl<  Svst—  y«atqr<« 

Th*  syst«B  should  bs  dsslgnsd  %rlth  the  following  ganoral  porfoza- 
anos  aaasTirss  in  aind: 

(1)  Ability  for  rsal-tias  oporabion  in  coaqplsx  aonochromatic 
scanas.  Tbis  conatrainb  will  ganarally  affact  taa^oral  and 
spatial  sanpllng  ratas,  as  wall  as  tha  coaplazity  and  ij^la- 
aantatioo  of  tha  rasulting  systaa  algorithns; 

(2)  Adaptability  to  tiaa-varylng  tazgat  and  (slowly  varying) 
background  panuatars,  in  particular,  thosa  ralatad  to  rota¬ 
tion  and  dilation  of  tha  targat;  and 

(3)  Miniisnn  probability  of  loss  of  targat  (LOT) ,  if  only  accord¬ 
ing  to  SOBS  loosaly  dafinad  critarion.  Latar  it  will  ba 
shown  that  a  sultabla  aaasura  might  ne  to  min  {E[(b-b}2]},  wh«re 
b  and  b  ara  tha  astiaatad  and  actual  targat  translational 
paranatars ,  raspactivaly. 

$ 

2.2  AlgorithM  Nacassarv  for  Succassfiil  Svstan 

Bafarring  to  Figura  2,  tha  winiiHosi  softwara  nacassary  for  a  suc- 
easaful  systm  ouiy  ba  subdividad  into  four  parts: 

(1)  A  targat/bsckgrooid  (T/B)  saparatlon  or  sagaantation  algo- 
rittasi,  idiiah  sagaants  tha  soana  by  classifying  pixals  (or 
groups  of  piasls)  as  assbars  of  aithar  tha  targat  or  back¬ 
ground  sats; 

(2)  A  ragistration  algoritha,  which  procassas  infozaation  froa 
tha  just-sagaantad  scans  as  wall  as  aaisory  inforaation  to 
gsnarata  raw  astiaatas  of  tha  targat  cantrold  (or  parhaps  da- 
viation  of  targat  cantrold  froa  sosa  nominal  valua) ; 
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Figure  2  Major  Tracking  Computer  Software  Components 
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(3)  A  txmckiag  filtar,  to  mlniaiao  th«  off  acts  of  aolsy  data,  In- 
•»et  T/B  sapazatlon,  partial  occlosion,  and  parlu^s  to  pre¬ 
dict  tha  taxgat  locatioo;  and 

(4)  An  oaarall  system  control  algorithm,  to  make  tha  major  system 
autoamtic  dacisions,  and  sx^arvisa  algorithm  intarzictian. 

Xt  should  ba  noted  that  tha  system's  overall  success  in  date  min- 
ing  the  target  position  is  highly  dapandant  upon  tha  success  of  algo¬ 
rithms  (1)  and  (2) ,  and  that  it  is  possible  that  both  these  functions 
may  ba  accoaplishad  via  one  algorithm,  e.g.,  a  template  matching  ap¬ 
proach.  Also,  in  this  report,  the  major  eaphasis  will  ba  on  tha  develop¬ 
ment  of  tha  first  two  algorithms ,  l.a.,  target/background  saparatiimi  and 
registration. 

2.3  System  Limitations  and  Assumptions 

Tha  foUcnfing  restrictions  and  assia^tians  are  posed  for  the  sys¬ 
tem  davelopad  hareiat 

(1)  Tb*  tracking  system  is  a  purely  passive  system,  i.a.,  it  pro¬ 
cesses  only  2-D  monochromatic  visual  information  resulting 
from  tha  ^ysics  of  tha  scene  enviranaiant  (  although  any  type 
of  information  containing  texture  should  be  satisfactory) . 

(2)  Tha  image  processing  algorithms  will  ba  generally  non-syntac¬ 
tic  in  approach,  thus,  according  to  Tanimato  [2],  tha  system 
ei^loys  "...image-driven  vieion..."  rather  than  "...knowledge- 
driven  vision. ..."  Ciis  assusqption  greatly  reduces  the  sys¬ 
tems*  need  for  artansive  a  priori  infozmaticn. 

(3)  The  system  algorithms  will  ba  designed  to  make  wmxijum  use  of 
tha  potential  advantages  of  charge  coupled,  device  (CCD)  im- 
plamantation.  Hhila  detailed  erplenaticns  of  the  utility  of 
CCD  devices  is  considered  elsewhere  [3-8],  basically  CCD's 
are  potentially  useful  for  the  following  reasons i 

(1)  NO  time-consuming  A/D  or  D/A  conversion  would 
be  required;  end 

(ii)  potential  processing  speed  is  predicted  to 
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be  in  excess  of  50  MHz  [9j. 

(4)  No  attempt  will  be  made  to  handle  tcurget  re-acquisition  or 
video  processing  in  cases  of  partial  or  temporary  tot2Ll  oc¬ 
clusion  . 

(5 )  The  target ,  even  in  the  case  of  coi!^>lex  scenes ,  possesses 
some  attributes  or  features  which  make  it  (automatically) 
distinguishable  from  the  background.  In  fact  (Section  5.3), 
it  will  later  be  assumed  that  the  target  possesses  a  signif- 
ic^ult  amount  of  texture. 

(6)  The  target  is  centered  in  the  camera  FOV  at  the  initial  or 
steucting  time  (perhaps  by  a  human  operator) . 

2.4  Comparison  of  Proposed  System  with  the  Humcui  Visual  System  (HVS) 
Obviously,  in  developing  "artificial  intelligence"  for  an  auto¬ 
matic  video  tracking  system,  one  standard  of  comparison  should  be  the 
human  visual  system  (HVS) .  While  it  is  clear  that  the  overall  operation 
of  the  HVS  system  is  not  totally  understood  [l],  certain  basic  princi¬ 
ples  of  operation,  e.g.,  several  of  the  Gestalt  laws,  may  provide  useful 
design  guidelines ,  and  possibly  may  be.  automated  in  real-time .  Appar¬ 
ently  HVS  operation  is  highly  syntactic  in  approach,  i.e.,  certain 
pieces  of  information  are  extracted  from  the  scene ,  and  together  with  a 
great  deal  of  a  priori  information  ("greunmar"  functions) ,  perhaps  ac¬ 
quired  over  a  span  of  many  years,  scene  analysis  proceeds.  Two  Gestalt 
laws  of  par'ticular  importance  are:  (1)  the  Gestalt  law  of  organization, 
which  roughly  says  that  some  areas  with  common  properties  (e.g.,  tex¬ 
ture,  pattern)  are  most  naturally  seen  as  a  unit;  and  (2)  the  Gestalt 
law  of  common  fate,  which  says  that  "...if  a  collection  of  parts  move  in 
unison,  one  tends  to  see  them  as  a  single  figure..."  [1].  Resecirchers 
have  made  significant  use  of  the  first  law,  e.g.,  in  texture  edge  detec¬ 
tion,  however  the  second  principle  heis  not  received  nearly  as  much 
attention,  probably  because  most  scene  amalysis  has  been  centered  on 
2-D,  time  invari^ult  functions.  It  will  be  shown  later  that,  for  moving 
targets,  automation  of  the  Gestalt  law  of  common  fate  may  be  extremely 
useful  for  segmentation  purposes. 
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In  additicai ,  the  HVS ,  having  a  biological  origin ,  is  highly  adaptive , 

both  in  real-time,  and  in  the  long  term.  The  automatic  system  under  consid-  1 

eration  here  should  also  be  adaptive,  however  it  is  probably  not  possible  >  | 

or  practical  to  autanate  long-term  adaptivity,  i.e.,  "artificial  evolution." 

Finally,  as  mentioned  adxjve,  the  HVS  utilizes  an  extremely  large  | 

amount  of  a  priori  information,  as  a  result  of  the  human  learning  process.  ^ 

Clearly  it  is  impracticcil  to  supply  the  system  considered  here  with  such  a  ■ 

large  training  set  of  scunples,  nor  would  it  be  practical  to  incorporate  mem-  ^ 

ory  sufficient  for  this  data.  i 


CHAPTER  III 


SUMKARZ  OP  PREVIOOS  MQBIC 

3.1  Prevloua  R«8»areh  Efforts 


Oua  to  the  extremely  current  nature  of  this  subject,  there  are 
only  a  handful  of  specific  references.  Perhaps  the  earliest  unclassi¬ 
fied  reference  is  that  of  Moskcwitz  [10]  in  1964.  P^er  outlines 

the  overall  problem  of  visual  target  tracking,  with  «>wpha«-i  g  on  succes¬ 
sive  scene  target  registration  and  the  effects  of  geometric  distortion. 
Two  possible  tenglate -matching  solutions,  i.e.,  correlation  and  moments, 
are  suggested,  and  brief  1-0  examples  of  these  methods  aure  included. 

?i  and  Grammes  [u]  present  an  analysis  and  simulation  of  a  video 
tracking  system  based  t^on  a  simple  centroid  and  second  moment  algo¬ 
rithm,  and  have  applied  this  c(»cept  to  extremely  simple  targets  and 
scenes  (white  backgxtxad,  blawk  target,  50x50  pixel  scene)  at  a  saa^ling 
rate  of  60  Hz.  This  ^proach  is  expected  to  yield  poor  perforaamce  with 
realistic  targets  and  scenes,  primarily  due  to  the  lack  of  an  adaptive 
target/background  sepauraticn  ad.gorithm.  A  similar  shortcoming  is  found 
in  the  simple  algorithms  of  Milstein  and  Lazicky  [l2 ] ,  Baker  and  McVey 
[13],  Woolard  and  McVey  [14],  and  Utbinski,  et  al.  [15]. 

Chow  and  Aggariwal  [16],  while  also  atssiming  unrealistically  sim¬ 
ple  targets,  consider  the  use  of  velocity  information  to  aid  in  teaiplate 
prediction  and  efficient  target  registratioo.  Matching  invariant  to 
translation  and  rotation  is  achieved  through  the  use  of  area  and  princi¬ 
pal  axis  as  features,  and  partial  occlusion  is  handled  via  a  sliq^le  ve¬ 
locity  based  prediction  algorithm  and  threshold  test. 

Gtao,  et  al.  [17]  devise  a  simple,  real-time  isg>lementable  algo¬ 
rithm  to  recognize  the  shapes  of  simple  objects  moving  horizontally  in 
the  FOV  of  a  TV  camera  through  a  single  projection  based  method  of  fea¬ 
ture  extraction.  Their  work  may  be  significant  in  that  they  establish 
an  object -dependent  Z/0  function  for  the  video  image  processor  (V.I.P,). 
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A  Similar  functicsn  is  proposed  by  BaJcer  and  McVay  [13],  and  is  used  in  a 
simple  microcomputer-based  tracJeing  system,  McVey  and  Mbolard  [14]  mav-, 
further  use  of  this  function  together  with  a  pertiurbation  method  to  de¬ 
termine  the  proper  control  signal  polarity  for  simple  image  tracking. 

Armstrong  [9]  couples  a  similar  1/0  control  function  with  a  simple 
signature-based,  horizontally  and  vertically  separaUble,  CCD-impleaent- 
able  image  processing  algorithm  to  form  a  closed  loop  control  system, 
and  provides  a  limited  stability  analysis  for  such  a  system. 

It  appears  that  the  most  current  emd  sophisticated  work  in  this 
area  is  due  to  Flachs,  et  al.  [13-24],  These  authors  propose  a  proto¬ 
type  real-time  video  tracking  system  which  uses  extensive  pajallel  pro¬ 
cessing  to  enable  tracking  in  512x512  pixel  scenes  at  a  rata  of  60 
frames/sec.  The  heart  of  this  system  is  a  Bayesian  pixel  classifier, 
aided  by  fuzry  set  logic  which  forms  a  binary  picture  subdivided  into 
the  target,  background  and  plume  regions.  Feature  extraction  is  accom¬ 
plished  via  binary  projections.  A  finite-state  synchronous  sequential 
is  used  to  generate  control  (pointing)  signals,  and  linear  N- 
point  polynomial  filters  are  used  for  smoothing  and  predicting  target 
trajectory  data. 

3.2  Shortcomings  of  Previous  Efforts 

The  cumulative  short caainga  of  previous  efforts  are  as  follows; 

(1)  The  assumption  of  unrealistically  simple  scenes,  and  there¬ 
fore  the  proposal  of  correspondingly  simple  algorithms,  most 
often  lacking  suitable  target/background  separation  or  regis¬ 
tration  algorithms.  This  generally  rules  out  the  successful 
use  of  these  algorithias  with  realistic  scenes; 

(2)  Unsuitability  for  real-time  operation.  Most  previous  work 
has  only  considered  algorithm  development,  and  most  often 
simulation  via  Fortran  prograsmdng .  little  attempt  has  been 
focused  on  the  real-time  implementation  of  even  the  simplest 
algorithms ; 

(3)  Inability  of  the  algorithms  to  handle  time-varying  scenes; 
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(4)  Inability  of  the  system  to  haiulle  targets  experience 

perspective  transformations,  in  particular  rotation  dila¬ 
tion;  and 

(5)  lack  of  any  type  of  a  scene  or  target  model  to  guide  algo¬ 
rithm  development.  This  probadaly  ea^lains  why  most  previotis 
efforts  were  es^irically  based. 


caapTER  rv 


MAXHEMAnCAL  FORHOLATION  OF  TBS  VIDEO  TRACEZNG  PROBLOl 


A  l09ic^U.  prerequisite  for  the  development  of  the  system  tracking 
algorithms  discussed,  in  Chapter  II  is  a  consideration  of  the  underlying 
processes  which  produce  the  system  timexvarying  camera  input.  There¬ 
fore  ,  in  this  chapter  a  general  3-0  mathematical  model  for  constrained 
scene-to-scene  target  motion  is  developed.  Using  this  model,  more  pre¬ 
cise  video  tracking  system  design  goals  may  be  formulated.  In  addition, 
a  closed-loop  block  diagram  for  the  overall  multidimensional  process  un¬ 
der  consideration  is  developed. 

4.1  Block  Diagram  and  Measurement  Variables 

A  first  step  in  the  development  of  a  mathematical  model  for  the 
system  described  in  Section  1.1  is  the  formulation  of  a  block  diagram, 
and  the  determination  of  measuraaent  and  control  variables.  Recalling 
the  Msumption  of  a  primary  discrete-analog  tracking  conguter,  it  is  ob¬ 
vious  that  the  system  will  operate  in  discrete  time,  i.e.,  the  system 
measurement  and  control  variables  will  be  obtained  at  discrete  times 
t  «  kT;  k  «  0,  1,  .  .  .N,  vdiere  T  is  the  temporal  sampling  interval. 
Also,  any  2-0  scene  at  time  kT  will  be  a  spatially  sampled  function,  for 
one  or  both  of  the  following  reasons: 

(1)  A  CCD-type  camera  generates  pixel  information  based  i^on  an 
NxM  discrete  sampling  lattice;  and 

(2)  The  discrete«?analog  processor  necessarily  operates  with  data 
at  discrete  Instants  (note  that  this  data  may  be  discrete  in 
time  or  apace) ,  therefore,  evn  if  the  scene  were  continuous 
in  space  it  would  be  necessary  to  spatially  sample  the  infor¬ 
mation  prior  to  processing.  More  will  be  said  about  sanpling 
in  Section  4.3.3. 

Looking  at  Figure  1,  two  points  are  noted: 

(1)  The  positional  control  variables  or  signals  (hereafter  de¬ 
noted  by  8(kT)  where 
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(4.1-1) 


and 

9a  (Ic^)  is  the  canera  mount  azimuth  control  at 
JcT; 

9Q(k7)  Iz  the  camera  mount  elevation  control  at 
JcT; 

and  z  (IcT)  is  the  camera  lens  zoom  control  at  time  (kT) ,  are  a 
function  of  the  measurement  veurieibles  or  raw  video  data  (here¬ 
after  denoted  by  p (i ,kT) ,  the  2-0  time  varying  picture  func¬ 
tion)  perhaps  at  times  (k-l)T,  (k-2)T,  etc. 

(2)  p[i,(k+l)T]  is  a  function  of  0(JcT)  as  well  as  other 

variables,  especially  the  maneuvering  target  trajectory,  as 
discussed  in  Section  4.3.2.  Thus,  it  should  be  expected  that 
the  overall  model  used  to  represent  this  sytem  is  of  the 
closed-loop,  feedback  type. 

Figure  3  shows  a  block  diagram  based  t^on  the  above  reasoning. 

The  upper  three  blocks  labelled  "T/B  Separation  and  Registration," 
"Tracking  Filter  and  Camera  Mount  Dynamics,"  emd  "Overall  Control"  cor¬ 
respond  to  those  algorithms  or  kncwn  constants  comprising  the  tracJcing 
conputer.  The  nun^rs  in  parenthesis  in  these  blocks  correspond  to  the 
algorithms  described  in  Section  2.2  and  Flg\ire  2.  The  lower  block,  la- 
Iselled  "Video  Process  X/0  Relationship,"  is  used  to  model  the  evolution 
of  the  picture  as  a  function  of  the  caamra  mount  controls  and  other  in¬ 
formation,  in  particular,  the  time-varying  scene  data  generated  by  the 
moving  target,  tlme-vary  background,  etc.  In  addition.  Figure  3  indi¬ 
cates  that  the  system  has  a  unique  feature,  namely  the  2-0  "link"  exist¬ 
ing  between  the  Video  Process  Z/0  Relationship  and  the  Tracking  Compu¬ 
ter..  The  Tracking  Computer  apparently  can  be  looked  i^)on  as  a  feedback 
controller,  with  the  not2ible  feature  that  it  processes  2-0  data  to  form 
1-0  control  signals.  The  block  corresponding  to  "Video  Process  I/O 
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Mulcldlmenslonal  Process 


Scalar  or  Vector 
1-D  Process 


Figure  3.  Basic  Closed  Loop  System  Model 
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BAlatlonshlp”  is  then  the  "plant,"  which  has  an  ejctemal  2-0  (uncontrol¬ 
lable)  input,  in  addition  to  the  camera  mount  control  Inputs. 

Obviously,  a  prerequisite  for  design' of  this  controller  is  the  de¬ 
velopment  of  a  suitable  mathematical  model  of  the  plant.  As  shown  in 
the  following  sections,  this  is  a  non-triviail  teisk,  and,  as  e:^cted, 
the  resulting  controller  structure  is  closely  related  to  the  chosen  mod¬ 
el  coiqplexity. 

4.2  TVfO  Coordinate  Systems 

The  target  dimensions  and  coordinate  values  in  the  target  (physi¬ 
cal)  coordinate  system  will  differ  appreciably  from  those  of  the  target 
image  in  the  camera  sensor  coordinates.  This  difference  is  a  function 
of  the  camera  lens  focal  length,  the  distance  of  the  sensor  from  the 
lens,  and  the  lens-target  distance  and  oiri-entation .  While  a  detailed 
description  of  this  situation  is  given  in  Duda  and  Bart  [25 ] ,  a  brief 
review  is  given  here  as  a  prerequisite  for  future  work. 

Assisaing  that  the  origins  of  both  the  2-0  sensor  coordinates  and 
the  2-0  target  coordinates  along  the  optical  axis,  that  the  lens- 
target  distance  eHong  this  axis  is  Icnown  and  neglecting  the  sign  inver¬ 
sions  due  to  the  (assumed  simple)  lens,  it  is  eaisy  to  show  that, 

7  *  -  7  *T  (4.2-1) 

where 

^  is  the  location  of  a  point  in  target  coordinates; 

X  is  the  corresponding  point  location  in  the  sensor 
coordinates ; 

s  is  the  cssmra-target  distance;  and 

f  la  the  sensor-lens  distance. 

Thus,  in  the  sensor  plane 

;  •  f*T  (4.2-2) 

or 


X  ■  Kd»r 


(4.2-3) 
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Typically,  is  very  small,  e.g.,  for 
f  a  50  mm 
z  ■  loo  m 

Kd  -  5x10"^ 

The  preceding  is  important  for  two  reasons: 

(1)  If  Kd  changes  due  to  a  change  in  z,  for  a  fixed  sensor  spa¬ 
tial  sampling  rate,  it  may  be  necesseury  to  adjust  Kd  via  f 
(zoom  lens  control)  in  order  to  maintain  proper  spatial  res¬ 
olution;  and 

(2)  The  tracking  algorithms  to  be  developed  will  estimate  target 
tremslation  in  sensor  units,  therefore,  in  designing  the  cam¬ 
era  mount  controls,  it  is  necessary  to  convert  this  informa¬ 
tion  into  physical  units  (Secticxi  5.4.7). 

For  the  remainder  of  this  work,  unless  otherwise  noted,  all  v^u:i- 
ables  will  be  referred  to  in  the  sensor  coordinates . 

4.3  Model  Development 

In  the  following  sections ,  models  for  the  static  scene  and  scene- 
to-scene  temporal  evolution  based  upon  the  cheuracteristic  function  and 
time-varying  affine  transform  cure  presented.  In  the  context  of  these 
models ,  the  video  tracking  problem  may  then  be  stated  as  a  (time-vary¬ 
ing)  affine  transform  parameter  estimation  problem. 

4.3.1  Basic  (Continuous)  Scene  Model  Formulation  (2-D) 

Following  the  ^lpproach  of  Nahi  [26],  it  may  be  shown  that  (for  the 
case  of  only  one  target)  the  scene  may  be  modelled  as  follows: 

p(x;t)  ■  f(x;t)X(x;t)  +  [l-X (x;t) ]b (x;t)  (4. 3. 1-1) 

where  the  following  3-0  functions  are  defined  as 

p(x;t)  ^  picture  function  intensity  at  sensor  location  x  and  time  t 

b(x;t)  ^  background  function  intensity  at  sensor  location  x  and 
time  t 

f(x;t)  ^  target  textural  function  intensity  at  sensor  location  x 


amd 


etnd  time  t 
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X(x;C)  •  scene  characteristic  or  replacement  function  value  at 
sensor  location  x  and  time  t,  where,  at  time  t 


flv  z 

X(x;t)  Q 


irtiere  target  exists  in  scene 
elsevrtMre 


Thus,  X(x;t)  is  the  binary  valued  fmctlon  which,  by  "turning  on" 
and  "turning  off"  the  target  and  background  functions,  respectively, 
gives  the  target  its  outline.  A  simple  example  of  scene  modelling,  us¬ 
ing  the  above  concepts,  is  shown  in  Figure  4.  Several  points  should  be 


noted: 


(1)  The  scene  model  is  in  general  a  non-linear  equation;  emd 

(2)  The  estimation  of  X(x;t),  in  a  complex  scene,  is 
a  very  difficult  problem. 

Several  other  useful  definitions  are: 

a(X;t)  A  f(;x;t)X(x;t)  (4.3.1-2a) 

n(x;t)  A  [1-X(x;t) lb(x;t)  (4.3.1-2b) 

«•  « 

so  that 


p(X;t)  -  s(x;t)  +  n(x;t)  (4.3.1-2C) 

Claaurly,  s(X(t)  is  the  function  that  is  to  be  tracked  over  the  in¬ 

terval  of  interest. 


4.3.2  Continuous  Scene  Temporal  Evolution  and  3-D  Mo<^«»i 
The  affine  transform  is  described  in  Appendix  (1).  For  tho  th- 
mainder  of  this  work,  it  will  be  assumed  that  target  perturbations  will 

dimensional  affine  transform  (further  restrictions  are  in  Section 

5. 4. 4.2).  This  means  that  target  motion  is  constrained  to  be  observed 
as  affine  motion  in  the  plane  perpendicular  to  the  optical  axis,  motion 
along  the  optical  axis  (dilation)  and  translation  in  the  plane  perpendic¬ 
ular  to  this  axis.  Clearly,  this  does  not  limit  target  motion  to  2-0, 
since  an  object  moving  simultaneously  to  the  right  and  towards  the  cam¬ 
era  may  be  thought  of  as  e^qseriencing  both  dilation  and  translation. 


X-0 


(c)  X(x) 


(d)  p(x) 

Figure  4  Basic  Scene  Fomulation  Using  'lodel 
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However,  perturbations  such  as  "roll"  of  a  target  whose  major  axis  is  in 
the  plane  perpendicular  to  the  optical  axis  would  be  prohibited.  Ease¬ 
ment  of  these  restrictions  and  extensions  of  the  model  are  discussed  in 
Section  4.3.4. 

In  the  continuous  (space  and  time)  case,  with  the  edsove  restric¬ 
tions,  the  target  "outline",  X,  must  be  perturbed  in  the  same  manner  as 
the  textural  function,  f,  therefore  the  taurget  function  evolves  tempor¬ 
ally  as 


where 

and 

and 


s{x;t2)  ”  s[A(At,ti)x  +  b(At,ti) ;ti]  (4. 3.2-1) 

*  f (Ax+bjti) *  X (Ax+b;ti) 

At  ■  t2-ti 
A  ^  A(At,ti) 

b  ^  b(At,ti)  (4. 3.2-2) 


are  the  time-varying  affine  transform  parameter  matrix  and  vector  re¬ 
spectively. 

At  this  point  also  assume  that  the  bacJcground  function,  b,  is  only 
slowly  time  v^ur/ing,  thus  over  a  period  of  several  temporal  samples  it 
is  approximately  const^mt,  i.e., 

for  t2-tj^  "small"  (this  is  covered  in  more 

detail  in  Section  5. 4. 6. 2) 


b(x!ti^)  »  b(x;t2) 


and  therefore 

n(x;t2)  -  [1-X(x;t2)]  b(x;t2) 

=  [1-X  (Ax+b;t2^)  ]  b(x;tl) 


(4. 3.2-3) 


Thus,  n  is  only  affected  by  changes  in  X. 

The  overall  scene  temporal  evolution  may  then  be  written  as 
p(x;t2)  »  f  (Ax+b;t2^)  X(Axtb;ti)  +■  [l-X  (Ax+b;ti)  lb  (x;ti)l  ' 


or 


p(x;t2)  *  s[Ax+b;tx]  +  [l-X (x;t2) ]b(x>ti)  (4. 3.2-4) 

A  blocic  diagram  of  target  temporal  evolution  for  two  successive 
scenes  is  shown  in  Figiire  5. 
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Figure  5  Constrained  Scene  Tennoral  Evolution 
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It  is  worthwhile  to  note  in  (4. 3. 2-4)  that  the  independent  variables 
are  x  and  t.  The  resulting  scene  at  t  =  t2  is  then  a  function  of  A(t) , 
b(t)  ,  and  p(x,t]^)  ■  Thus,  A(t)  and  b(t)  are  used  in  this  model  to  control 
the  relative  position,  size  and  orientation  of  the  target  in  successive 
scenes.  In  general,  these  parameters  will  be  functions  of  both  the  camera 
mount  controls  and  the  target  trajectory.  As  shown  in  the  following  sec¬ 
tions  ,  a  method  for  the  estimation  of  these  parameters  forms  the  basis  of 
the  tracking  computer. 


4.3.3  Modifications  Due  to  Sampling 
and  Discrete  3-D  Model  Formulation 

While  tne  prec»^ding  model  was  developed  in  continuous  time  and  space, 
it  is  apparent,  from  Section  4.1,  that  a  discrete  time  £md  space  model 
is  needed.  Fortunately,  the  conversion  of  the  previously  developed  model 
to  discrete  parameters  is  very  sin^jle  and  straightforward.  It  is  only  nec¬ 
essary  to  assume  that  the  3-D  continuous  picture  funciton  p(x,t)  is  sampled 
spatially  and  temporally  by  a  "3-D  sampler,"  i.e.,  an  ideal  3-D  impulse 
function  of  the  form 


6  (x-i ,  t-kT) 

where 


UJ 


X2^,X2eR 


i»je  t 


(4. 3. 3-1) 


(4. 3. 3-2) 


A  is  the  spatial  scunpling  interval  (assumed  equal  in  the 

horizontal  and  vertical  directions)  and 

T  is  the  temporal  sampling  interval ,  yielding 
00  00  00 

p*(i;kT)  p (x;t) 5 (x-i ,t-kT)  (4, 3. 3-3) 

i*  — ooj*  — 0oJc»  — ® 

The  mathematical  considerations  2uvd  approximations  involved  in  such 
a  formulation  are  analogous  to  the  well-known  1-D  case,  e.g.  ,  we  assiome 
that  the  di3crete-^ulalog  tracking  computer  incorporates  a  (3-D)  zero-order 
hold  device. 

» 

Thus ,  at  the  output  of  the  ZOH  we  have  the  sampled  picture  function 
p(ij  kT) ,  and  the  corresponding  3-D  discrete  model  temporal  evolution  may 
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siaply  b«  obtaijed  from  (4. 3,2-4)  as: 

p[i;ac+l)Tl  «  f [A(kT)i+bOcT);kT]X[A(]cT)i-**(lcT);JcTj 

+  [l».X[AOrt:)i+bacT)];)cT>)(i;)eT)  (4. 3. 3-4) 

In  the  following  sections  It  will  often  be  useful  to  switch  be¬ 
tween  the  discrete  and  continuous  model  formulations ,  and,  In  the  context 
of  the  model  developed  here ,  this  will  not  result  In  any  mathematical  or 
practical  difficulties.  It  should  be  noted,  however,  that  In  general, 
many  digital  picture  processing  operations  are  not  ’’reversible"  In  the 
sense  that  conversion  between  the  discrete  and  continuous  models  and/or 
parameters  Is  straightforward,  or  even  possible.  An  excellent  example  of 
this  Is  the  attes^ted  correction  of  a  discretized  picture  which  has  un¬ 
dergone  perspective  distortion  (c.f.  [l]}. 

Finally,  the  effects  of  the  sampling  parameters  A  and  T,  with  re¬ 
spect  to  Fourier  sampling  constraints,  derivative  estimation,  and  the  va¬ 
lidity  of  Taylor  series  approximations  aire  discussed  in  Sections  5.4. 6,1 
and  5. 4. 6. 2. 

4.3.4  T.iTn-{  tations  and  Extensions  of  Model 

While  many  of  the  constraints  and  assis^ptlons  associated  with  the 
previously  developed  model  are  not  overly  restrictive.  It  Is  lUcely  that 
once  successful  solutions  based  upon  this  model  have  been  developed, 
these  restrictions  will  be  grzidually  eased  and  new  solutions  (hopefully) 
found,  until  the  final  model  ^ulequately  represents  a  very  wide  class  of 
real-world  situations.  For  this  reason,  some  obvious  extensions  of  this 
model  are  presented. 

As  mentioned  In  Section  5.4. 4. 2,  whenever  A  does  not  represent 
strictly  rotation  and/or  dilation,  the  target  Is  actually  being  observed 
through  a  different  viewing  angle.  Practically,  It  should  ailso  be  ex¬ 
pected  that  new  target  features  should  appear,  particularly  airoimd  the 
target  edges.  One  possible  way  to  update  the  model  C4,3.3-4)  to  handle 
this  information  might  be  to  model  it  as  an  additional  fimctlon,  e.g, 

p[i.  (k+l)T]  -  [f  CA(kT)i+bOcT);  JcT)+e(kT)  ]  •X[A(]cT)i+b(kT);  icT] 
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+  [l-X(A{kT)i+b(kT) ;kT]b(i;kT)  (4. 3.4-1) 

where  e (kT)  is  primarily  "edge  oriented." 

Closely  associated  with  the  above  extension  is  that  of  handling 
"roll"  of  a  target  whose  major  axis  is  in  a  plane  perpendicular  to  the 
optical  axis.  In  this  case  it  should  be  noted  that  the  eissumption  in 
(4. 3.2-1)  is  no  longer  valid,  i.e.  ,  A  is  no  longer  perturbed  in  the  same 
manner  as  f.  In  fact  only  f  is  perturbed  in  this  case,  and  the  scene  tem¬ 
poral  evolution  model  will  need  to  be  adjusted  accordingly. 

The  model  may  be  adapted  to  handle  multiple  targets  by  defining 
multiple  target  and  characteristic  functions,  e.g.,  f^,  f2  and  A2. 

If  these  targets  are  allowed  to  overlap  (i.e.,  pass  in  front  of  one  an¬ 
other  in  the  FOV  )  it  will  be  necessary  to  prioritize  the  Ai  functions,  in 
order  to  insure  that  at  any  scene  point  only  one  Ai  is  non-zero.  Also 
note  that  this  adaptation  permits  the  handling  of  occlusion. 

Another  important  modification  of  the  model  might  be  to  permit 
time-varying  scene  illumination  or  brightness  variations.  This  may  be 
accomplished  in  a  rather  straightforward  manner  by  simply  multiplying  the 
target  and  background  functions  by  suitable  time-varying  constants. 

4.4  Review  of  Previous  Efforts  Using  Model 

The  vast  majority  of  the  previously  cited  research  ignores  the 
scene-to-scene  temporal  link  modelled  in  Section  4.3.2,  and  discussed  in 
Section  5.1.  In  the  context  of  the  model  developed  in  Section  4.3.1  and 
the  references  cited  in  3.1,  some  approaches  have  been  (with  teirporal  in¬ 
dices  omitted  for  sin^ilicity)  : 

(1)  Assume  b(x)  =  0  V  x  so  that  essentially  an  all-white  (or 

black)  background  is  present  and  therefore  a  T/B  separation 
algorithm  is  not  necessary.  Then  use  simple  algorithms  to  es¬ 
timate  the  centroid  of  s  (x) .  (e.g.,  Yi  and  Grommes  [ll], 

Baker/McVey  [13],  Woolcird/McVey  [14],  Uno  [l7]). 

Edge  detection  for  T/B  separation:  Constrain  f(x)  and  b(x) 
and  estimate  spatial  derivatives  by  differentiation  of  equation 


(2) 


1 


(4. 3. 1-1)  in  both  the  horizontal  amd  verticaLl  directions  as 


follows: 


where 


j  -  1,2 


C4,4-l) 


(4.4-2) 


Then  assume  that  both  the  target  amd  background  have  very  little 
texture  {see  Section  5.2)  and  that  the  major  adarupt  changes  in  pixel  in¬ 
tensity  in  the  scenes  are  due  to  the  taurget  edges.  Thus, 


3Xj 


which  leauis  to 


f(x)-b(x)  =  K 


so, that  (4.4-1)  becomes 


ifiiiL  =  ^ 

3xj  3xj  ^ 


(4.4-3) 


Thus,  spatial  derivatives  aire  estimated  (perhaps  by  a  simple  dif¬ 
ference  equation)  and  it  is  assirsed  that  x's  where  is  "large" 

are  points  on  the  target  edge.  It  should  be  obvious  tha^  if  the  preced¬ 
ing  assua^tions  are  not  valid  (e.g.,  in  realistic  or  complex  scenes)  , 
that  it  will  be  vary  difficult  to  estimate  the  target  contour  (e.g., 
Chow/Aggarwal  [16]). 

(3)  Thresholding  for  T/B  separation: 

AssTime 

E{f(x)}iKi 
Kj,>E{b(x)  }^2 
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Then  design  a  pixel  classifier  such  that; 

^  is  a  target  pixel  if  pC5c)>Ki 
X]^  is  a  background,  pixel  otherwise . 

^^is  approach  is  also  t^estionable  in  realistic  scenes  ^  since 
it  is  likely  that  both  the  target  and  background  functions 
will  have  significant  intensity  variations,  and  therefore,  if 
this  approach  is  taken ,  the  resulting  segmented  scene  will 
have  "measles'*  [1] , 

(4)  Statistical  segmentation;  Assume  that  f(x)  and  b(x)  may  be 
distinguishable  by  seme  statistical  test  (variance,  pdf  esti¬ 
mate,  etc.).  Then,  using  this,  test,  classify  pixels,  or 
grot^s  of  pixels,  as  either  originating  in  f(x)  or  b(x)  (e.g., 
Plachs  [18-24]  ;  "textTire"  edge  detection)  . 
xn  summary ,  approaches  (1-3)  are  of  very  limited  use  in  typical 
scenes,  and  the  accuracy  and  efficient  ia?>lemantation  of  (4)  is  question¬ 
able. 


4.5  Problem  Statement  Using  Model 
Assumption  6  in  Section  2.3  assumed  that  the  target  is  centered  in 
the  camera  FOV  (perhaps  by  a  human  operator)  at  t  »  kT  ■  0.  Therefore, 
using  the  previously  developed  model,  and  assuming  that  the  effect  on  the 
scene  due  to  camera  mount  controls  is  Jenown  (this  is  addressed  in  Section 
5.4.7),  (4. 3. 3-4)  indicates  that  any  sequence  of  M  scenes, 

{p(i>kT) };  k  -  1,2,  .  .  ,M 

is  related  by  a  seqiience  of  6M  affine  parameter  variables  A(kT)  ,  b{kT); 
k  *  1,2,.  .  .M. 

Recall  from  ^pendix  1  that  b(kT)  represents  the  target  transla¬ 
tion  during  the  sampling  interval  Ci.e.,  the  period  from  t  ■  kT  to 
t  ■  (k+l)T).  If  b(kT)  were  known  over  the  range  of  k  of  interest,  the 
tracking  problem  would  be  simplified  consideradily ,  since  this  data  could 
be  combined  with  other  (assumed)  known  system  parameters  (Section  5.4, 7) 
to  generate  9a(kT)  and  3g (kT)  over  this  interval.  Similarly,  if  A(kT) 
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were  also  known,  the  formulation  of  z{kT)  would  be  simple.  These  para¬ 
meters  are  not  known,  however;  therefore  the  basic  image  tracking  prob¬ 
lem  may  be  stated  as : 

Given  p(i;0)  amd  the  M  succeeding  scenes,  p(i;kT) ; 
k  =  1,2,.  ,  .M,  generate  "good"  estimates  of  A(kT) 
and  b(kT);  k  =  1,2,.  .  .M,  and  use  these  estimates 
to  form  0(kT);  k  =  1,2,.  .  .M,  such  that  the  target 
tends  to  remain  centered  in  the  caimera  FOV,  and  (per¬ 
haps)  approximately  the  same  overall  size. 

Thus,  the  algorithms  for  the  estimation  of  A(kT)  and  b(kT)  form 
the  basis  of  the  tracking  computer.  A  composite  model  for  the  overall 
video  tracking  problem  is  shown  in  Figure  6.  While  there  probably  exist 
a  large  number  of  possible  solutions,  the  majority  of  the  remainder  of 
this  work  involves  the  novel  solution  of  this  problem  using  data  seunples 
from  two  successive  scenes  and  an  assumption  concerning  scene -to- scene 
temporal  dependence.  This  solution  has  some  very  importamt  euid  desirable 
properties,  in  particular,  the  potential  for  real-time  operation  and 
relatively  simple  CCD  implementation.  Alternate  solutions  based  upon 
this  model  are  discussed  in  Chapter  VII. 


28 


CHAPTER  V 

DEVELOPMENT  OP  THE  TAYLOR  ggPTffg 
VIDEO  IMAGE  PROCESSOR  CTSVIP)  ALGORITHM 


This  chapter  presents  a  novel  and  Ideally  CCD-inpleoentable  solu* 
tion  to  the  video  tracking  problem  as  posed  in  Section  4.5,  By  assuming 
a  sufficiently  fast  system  temporal  sampling  rate,  it  is  shown  that  the 
scene-to-scena  temporal  evolution  modelled  in  Section  4,3.3  may  be  ap¬ 
proximated  using  a  linear  model.  cCD  iaglementable  estimation  algorithms 
are  then  developed  for  this  model  to  provide  estimates  of  target  transla¬ 
tion,  rotation  and  dilation, 

5*1  Scene  Temporal  Dependence,  Temporal  Sampling  Rate  and 
Algorithm  Complexity,  and  Time/Space  Computational  Trade-Offs 

The  3-D  model  develc^d  in  the  previous  section  clearly  shows  the 
scene-to-scena  tea^wral  dependence  ejected  when  viewing  a  moving  target. 
It  is  ejected  that  as  the  tang>oral  sampling  rate  increases,  this  scene- 
to-scene  tea?>oral  dependence  also  increases,  i.e.,  as  T-*-0  there  is  very 
little  change  in  siiccessive  scenes.  Conversely,  for  a  relatively  large 
tea^ioral  saa^iling  interval,  T,  it  is  ejqjected  that  consecutive  scenes 
would  have  relatively  little  information  in  connon,  and  therefore  the 
problems  of  extracting  target  features  invariant  to  e:^cted  affine  per¬ 
turbations  and  the  estimation  of  these  perturbations  become  significantly 
more  difficult,  and  lead  to  increasingly  con^plicated  algorithms.  There¬ 
fore,  if  T  nay  be  (diosen  sufficiently  "small"  (this  Is  defined  more  pre- 
^  Section  5. 4, 6. 2),  the  above  reasoning  may  be  es^loyed  to  design 
a  set  of  efficient  and  accurate  tracking  algorithms,  hereafter  referred 
to  as  the  Taylor  Series  Video  Image  Processor  CTSVIP)  algorithios. 

The  concept  of  system  time/space  computational  tradeoffs  is  also 
noteworthy.  As  described  in  Blaauw  [27],  the  total  system  cost  to 
achieve  specified  data  processing  is  a  function  of  "time"  and  "space," 
where  "time"  is  expressed  as  a  fxanction  of  the  delay  times  of  the  physical 
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components  that  realize  the  desired  logic  function,  and  "space"  refers 
to  the  number  of  physical  components  en^jloyed.  For  the  system  consid¬ 
ered  here,  it  is  clearly  desirable  to  minimize  the  temporal  cost,  in  or¬ 
der  to  achieve  real-time  operation ,  therefore  it  will  be  assumed  that 
the  spatial  cost  of  the  system  (i.e,,  the  number  of  components  per  IC 
chip  cuid  the  total  number  of  chips)  is  relatively  insignificant.  Thus, 
the  system  should  be  designed  to  operate  with  a  temporal  sampling  rate 
sufficiently  high  to  enable  real-time  operation,  and  medce  maximum  use  of 
computational  complexity  in  "space."  The  TSVIP  algorithms  to  be  pre¬ 
sented  make  significant  use  of  parallel  processing,  and  also  inherently 
enable  temporally  efficient  "pipelining,"  since  smapled  information  from 
the  present  scene  may  be  processed  while  the  next  scene  is  being  sampled 
simultcuieously. 


9f  (^)  ^ 

5.2  Target  Texture  Definition  and  Relationship  to 

dX 

Image  "texture"  is  a  perceptual  phenomenon.  Often  this  term  has 
been  used  to  denote  some  particular  ad-hoc  measure  of  the  spatial  vari¬ 
ations  in  light  intensity  of  an  image,  e.g.  ,  a  standard  bar  pattern  would 
be  thought  of  as  a  highly  textured  pattern.  This  research  views  texture 
from  a  more  mathematical  point  of  view,  particularly  when  considering 

target  texture.  By  definition,  the  textured  tcurget  function  is  repre- 

,3f (X;t) 


sented  by  f(x;t),  where 


3x 


for  all  t  of  interest.  Thus,  the  greater  the  value  of 


is  nonzero  for  some  set  of  xj^'s,  and 

3f  (Xkt)' 


3x 


at  a 


particular  x,  and  the  larger  the  set  of  Xj^'s  where  this  derivative-based 
function  is  nonzero,  the  more  "textured"  the  target  is  said  to  be.  Tex¬ 
ture  is  thus  a  target  spatial  feature.  Eqviivalently ,  a  teurget  is  tex¬ 
tured  if  the  2-D  function 


[f (x;t)]t=tp  ^  F(u) It-tp  (5.2-1) 

has  nonzero  components  at  spatial  frequencies  other  than  u  *  0. 

Texture  has  been  shown  to  be  very  useful  for  scene  segmentation, 
however,  in  the  work  which  follows  it  will  be  shown  that  target  texture 
may  also  be  used  to  aid  tracking  algorithms .  Thus ,  it  is  necessary  to 
assume  that  the  target  of  interest  is  textured  (perhaps  even  after  fil¬ 
tering  and  preprocessing) .  This  is  not  an  overly  restrictive  assump- 
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tion,  however,  for  two  reasons: 

(1)  Most  complex  tairgets  of  interest  exhibit  this  feature;  and 

(2)  The  class  of  textureless  targets,  which  is  typically  assumed 
in  previous  research,  enedsles  much  simpler  processing  algo¬ 
rithms  for  both  segmentation  emd  registration,  therefore  they 
are  not  of  interest  in  the  present  work. 


5 . 3  Taylor  Series  Expansion  of  Model 
amd  Corresponding  Scene  Temporal  Evolution 
In  the  following  two  sections  the  scene  temporal  evolution  model 
(4. 3. 3-4)  is  approximated  using  a  truncated  Taylor  series  expainsion.  By 
restricting  the  tracking  processor  input  to  data  from  the  target  inter¬ 
ior  (temporarily  assuming  T/B  separation  has  been  accomplished) ,  the 
model  may  be  simplified  even  further,  and  this  simplification  enables 
the  affine  parameter  estimation  techniques  of  Section  5.4. 

5.3.1  Direct  Approximiation  for  Small  A,  ^ 

Recall  the  assumption  that 

b(x;ti)  =  b{x;t2)  for  t2-t]^  "small."  Also  note  from  the  for¬ 
mulation  of  scene  evolution  in  (4. 3.2-4)  that  temporal  changes  in  f  and 
X  are  only  due  to  the  time  varying  A  and  b  pcurameters ,  therefore 


3f (X;t) 
3t 


3(X;t) 

3t 


0  Vx  and  Vt 


We  can  expand  s(x;t2)  and  X(xjt2)  about  x  and  keep  only  the  first  order 
terms.  Dropping  the  temporal  indices  for  simplicity: 


s (Ax+b)  ;  s  (x)  + 


,3s  {?),T 


(A-I)X  + 


(5. 3. 1-1) 


where 

clearly 

and 


»  s(x)  + 


AAx  +  ( 


3s  (X) 
3x 


■)Tb 


AA(At,ti)  «  A(At,ti)  -1 
A(0,t)  *  I 
b(0,t)  -  0 


(5. 3. 1-2) 
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so  that 

AA(0,t)  =  0  . 

Similarly 

X(Ax+b)  *  X(x)  +  ^  )'^AAx  +  (-— ")'^b 

-  '  '  X  '  X  ' 


(5. 3. 1-3) 


Two  remeurks  are  in  order: 

(1)  Clearly  the  above  equations  are  only  valid  for  | | (A-I)x+b| | 
"small."  This  will  be  addressed  in  Section  5. 4. 6. 2;  and 

(2)  Due  to  the  binary  nature  of  X(x;t),  s<xne  modification (s)  will 
be  necessary  to  handle  (-1^)^.  As  will  be  shown  later,  this  is 

dX 

not  a  serious  problem. 

Thus ,  with  the  aUhove  assumptions ,  it  is  easy  to  show  that  the  total  scene 
evolves  from  (4. 3.2-1)  as; 

p(X!t2)  :  s(x;tj^)  -i-  ]'^(AAx-t-b) 

+  b(x;ti)  [1-X(x;ti)  -  (9\^>-tl))T(4Ax+b)]  (5. 3. 1-4) 


Recalling  s  *  fX,  expanding 


fas  (^;t 

L 


ll 


yields 


p{x;t2)  :  f  (x;ti)X(x;ti)  +  [  (  ) '^x  (x;  ti) 

dX  • 

+  f(x;ti)  (■^^^|i^)’’](AAx+b) 

+  b(x;ti)  [1-X{x;ti)  ]  -  b(x;ti)  (■— ^*^l^'^(AAx+b)  (5. 3. 1-5) 


so 

there 

are  3  primary  scene  regions ; 

(1) 

X  = 

<t>  -  Q 

(Interior  of  target) 

(2) 

X  =■ 

0,  (|i)  .  0 

(Bacicground) 

(3) 

X  - 

0  or  1;  (|^)  -  "large" 

(Target  contour) 

For  the  remainder  of  the  discussion  in  this  section,  assume  the 

3X 

system  is  only  operating  on  points  where  A  »  1  ^md  -r—  “  0.  Thus,  only 

oX 

target  texture  points  are  being  considered  and  the  segmentation  problem 
temporarily  is  assumed  solved.  The  topic  of  segmentation  is  covered  in 
deteul  in  Section  5.4.5.  The  T/B  separation  algorithm  provides  an  esti- 
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mate  of  p(x;t)  for  A(x;t)  =>  1  (i.e.  r  f(x;t)  on  the  target  interior).  De¬ 
note  this  estimate  by  Pg(x;t)  where  Pg{x;t)  =  p(x;t)  Vx  where  X(x;t)  =  1. 
The  remaining  system  is  therefore  "neeursighted"  in  that  it  operates 
strictly  on  points  in  the  interior  of  the  target. 

Now  define  the  scene  difference  function  based  upon  target  tex- 

tural  points  as ; _ 

d{x;t2)  =  Ps(x;t2)  -  (5. 3. 1-6) 

Recalling  past  assumptions,  it  is  easy  to  shew  that: 

d(x;t2)  '  (|^^''*^^^^rAA(At,ti)x  +  b{At,ti)]  Vxe{Ri}  (5. 3. 1-7) 

where  {R^ }  is  the  set  of  all  points  on  the  interior  of  both  the  original 
and  perturbed  targets. 

Remark — This  prob^d3ly  seems  obvious  intuitively,  however  the  preceding 
work  is  useful  in  that  it  enables  handling  the  more  general  case,  i.e., 
we  can  study  the  effect  of  inexact  segmentation  on  the  algorithms  to  be 
developed  (Section  5. 4. 6. 3). 

If  (5. 3.1-7)  holds,  then  a  very  nebulous  3-D  problem  may  be  reduced 
to  one  of  1-D  (time-varying)  parameter  estimation.  A  note  should  be  made 
that,  while  the  preceding  derivations  were  for  the  continuous  case,  these 
ideas  also  apply  to  the  discretized  version,  as  explained  in  Section 
4.3.3.  Before  proceeding  to  estimation  considerations,  it  is  useful  to 
consider  model  temporal  evolution  from  a  different  (but  equivalent)  point 
of  view. 

5.3.2  Using  the  Scene  Temporal  Differential  Equation 
and  Difference  Approximations 

Recall,  from  (4. 3.2-1)  that,  for  the  target  textural  function  alone 
(i.e.,  where  X(x;t)  =  1) 

f(x;t2)  =-  f  [A(t2-ti^,tj^)x  +  b(t2-ti,ti)  ;ti]  (5. 3. 2-1) 

Expanding  (5. 3.2-1)  in  a  Taylor  Series  expemsion  and  retaining  only  the 
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I 


I 


linear  terms  yields  (temporal  indices  are  understood) ; 
f(Ax+b)  =  f(x)  +  (^f^)^(AAx+b) 

aX  ^ 

SO 


f(x;t2)  - 

By  definition 

df (X;tl) 
dt 


f(x;ti)  =  [AA  (At ,  ti)  X  +  b(At,t]_)] 

A  lim  f (X;t,)  -  f (X;ti ) 

^  A«-  -f 


lim 

At^ 


(^^^f^^)'^[A(&t,ti)-I)x  +  b(At,ti)] 
At 


(5. 3.2-4) 


(5, 3. 2-5) 


Obviously  (5. 3.2-5)  goes  to  0/0  in  the  limit,  so  we  can  apply  L' Hospital's 
rule ,  yielding : 


df (^;t)  _  ,af(X;t)?f  dA(T)  db(t)i 

dt  '  3?  ^  ^  dt  '  dt  ^ 


(5. 3. 2-6) 


Equation  (5. 3.2-6)  clearly  shows  that  target  textural  function  tem¬ 
poral  and  spatial  evolution  are  linked,  the  "link"  being  the  temporal 
P^^tial  derivatives  of  the  affine  transform  parameters.  It  is  this  link 
or  dependence  which  has  been  ignored  in  past  efforts,  euid  which,  when 
carefully  exploited,  leads  to  the  novel  tracking  algorithms  presented  in 
this  work.  Thus,  in  the  context  of  the  previously  developed  model,  scene 

temporal  evolution  is  governed  by  and 

9t  3t  3x 

Furthermore,  it  is  easy  to  show  that  (5. 3.2-6)  also  leads  to 
(5. 3. 1-7),  by  approximating  derivatives  in  (5. 3.2-6)  as  differentials: 


I 


J 


Af (X;ttAT)  ;  .3f (X;t)  T  AA(t)  Ab(t) 
At  3x  '  At  '  At 

.Multiplying  by  At,  recalling 

At  =  t2~t2^ 


(5. 3.2-7) 


and 


4A(At,tx) 
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»  A(At,ti)-A(o,ti) 

*  A(At,ti)-I 

and  defining 

Ab(ti)  “  b(At,ti)-b(o,t]^) 

=  b(At,ti) 

we  see  that 

Af(x;t2)  =  (  ^')  [(A(At,t].)-I)  x  +  b(At,ti)]  (5. 3. 2-8) 

and  from  (5. 3. 1-6),  since  we  are  only  considering  the  region  where  X  =  1, 
d(x;t2)  *  d(x;ti+At) 

=  f(x;ti^+At)  -  f(x;ti)  (5. 3. 2-9) 

=Af  (x;tj^+At) 

then,  using  (5. 3. 2-8), 

d(x;t2)  =  (^^^|^^^)^[{A(At,ti)-I)x  +  b(At,ti)  ]  (5.3.2-10) 

which  is  identical  to  the  previously  derived  (5. 3.1-7)  for  all  points  on 
the  target  interior. 

5 . 4  Reduction  to  1-D  Model  and  Solution  Approaches 
In  this  section,  the  3-D  model  of  Section  5.3  is  reduced  to  a  1-D 
model,  and  it  is  shown  that  the  video  tracking  problem  may  be  cast  as  a 
time-varying  affine  trcuisform  parameter  vector  estimation  problem.  Sev¬ 
eral  possible  solution  approaches  are  then  presented,  in  particular,  one 
CCD-implementable  solution  based  upon  the  formation  of  a  pseudoinverse  of 
an  Nx4  parti tional  matrix  is  discussed  in  detail. 

5.4.1  Scene  Difference  Function  with  Assumed  Segmented  Scene, 

Reduction  to  1-D  Problem  cuid  Statistical  Sampling 
Recall  (5. 3. 1-6) ,  i.e. , 

d(x;t2)  =  Ps(x;t2)  -ps(x;ti)  (5. 4. 1-1) 

In  the  actual  discrete  time  system,  d(x;t2)  is  not  measured,  but 
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rather  the  sampled  version  of  this  function,  i.e.,  d(i;(k+l)T)  (Section 
4.3.3) •  This  scene  difference  function  may  then  be  "re-sampled”  (in  a 
statistical  sense);  yielding  an  Nxl  1-D  vector,  hereafter  denoted  as 
d((k+l)T)  or  more  simply  d  in  the  remaining  discussion.  Thus, 

d((k+l)T)  -  Cd(xi;  (k+l)T)]  (5. 4. 1-2) 

where 

i=l,  2,  .  .  .  ,N  (5. 4. 1-3) 

is  the  vector  x  at  the  i^^  sampling  point.  An  importeint  point  here  is 
that  the  Xj_'s  need  not  be  consecutive,  i.e.,  with  some  limitations,  sam¬ 
ples  in  both  the  horizontal  cind  vertical  directions  may  be  s)d.pped  since 
eventually  an  overdetermined  set  of  linear  equations  will  be  formulated 
and  therefore  only  a  set  of  N  "good"  statistical  samples  is  necessary. 


?i  = 


^i2 


5.4.2  Matrix  Formulation 


Defining 


AA(kT) 


aj^l  (kT) 
a21  (kT) 


and 


b(kT) 


bi (kT) 
b2  (J^T) 


ai2  (kT,' 
322  (>^T) 


and  medcing  the  following  additional  definitions 


9f (Xi;kT)  T 

3? 


[fil  fi2] 


and 


ail  (kT)“ 

ai2 (kT) 

ap  (kT) 

321  (kT) 

322  (>«T) 

bi (kT) 

b(kT) 

b2 (kT) 

(5. 4. 2-1) 


(5. 4. 2-2) 


(5.4.2-3a) 

(5.4.2-3b) 
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(5.3. 1-7),  (5. 4, 1-2)  and  (5. 4. 1-3)  and  tha  abov*  equanions  aay  be  used 
to  show  (dropping  the  taiaporal  index  for  aiapUcity)  ; 


d(^) 


'U^il 


*12^11  *il^U  *12*12: 


*il  *12] 


*ul 


*12 


*21 

*22 


^1 

b2 

(5, 4. 2-4) 


In  conaiderlng  the  N  samples,  the  following  matrix  equation  Is  obtained: 


*11*11  *12*U  *11*12  *12*12  *11  *12" 

*11' 

d(^) 

*21*21  *22*21  *21*22  *22*22  *21  *22 

I 

*12 

e 

• 

*21 

« 

.  1 
:  : 

*22 

bl 

b2 

d(xj|) 

*Ml*Ni  *(l2*ttL  **I1*N2  *M2*N2  *Nl  *N2 

Written  more  compactly,  (5. 4. 2-5)  yields: 


d[(Jc+l)Tl  -  D()cT)a()cT) 
where  d  is  ilxl,  0  is  !lx6  and  a  is  6x1. 


(5. 4. 2-5) 


(5. 4. 2-6) 


For  future  purposes  It  is  also  convenient  to  partition  0,  yielding 


D  -  [P  i  G] 


(5, 4, 2-7) 


where,  from  (5, 4, 2-5),  G  la  an  11x2  matrix  of  unweighted  spatial  deriva¬ 
tives,  and  P  Is  an  Nx4  matrix  of  weighted  spatial  derivatives^ all  eval¬ 
uated  at  t  -  ]cT.  It  Is  assumed  that  the  elements  of  G  and  P  are  known 
(or  calculable)  and  more  will  be  said  about  this  in  Section  5. 4. 6.1. 
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5.4.3  The  Basic  Estimation  Problem 
Before  making  a  general  statement  concerning  (5. 4. 2-6)  ,  it  is  de 
sirable  to  formulate  some  objectives. 

Recall  that  a{kT)  was  defined  in  (5. 4. 2-3)  ass 


a(kT) 


'aii(kT) 

I 

a.12  OtT) 
a2i (kT) 

a22  (Jt") 


|ap(kT) 


(5.4. 3-1) 


I  b(kT) 

_  b2  {IcT)J  L 

The  video  image  processing  system  may  be  interesting  in  estimating 
the  translational  vector,  ^(kT),  i.e.,  the  last  two  elements  of  a,  but 
a  knowledge  of  the  entire  a (kT)  vector  may  also  be  useful,  for  several 


reasons : 

(1)  If  the  system  experiences  dilation,  it  may  be  highly  desirable 
to  adjust  the  camera  zoom  lens  control  cind  a  good  control  sig¬ 
nal  could  be  derived  from  ap;  and 

(2)  An  estimate  of  the  entire  a  vector  may  also  be  used  to  update 
the  system  segmentation  algorithm  to  insure  only  target  inter¬ 
ior  textural  points  are  sampled  (Section  5. 4. 5.1). 

Thus  there  are  several  different  approaches  which  could  be  taken  at  the 
present  time : 

(1)  Given  (5. 4. 2-6),  generate  a  "good"  estimate  of  a(kT),  with  no 
constraints  on  AA(kT)  other  than  those  necessary  to  ensure  the 
Taylor  series  expansion  holds.  This  approach  is  covered  in 
Section  5. 4.5.1, • 

(2)  Given  (5. 4. 2-6),  generate  a  "good"  estimate  of  only  b(kT). 

This  topic  is  covered  in  Sections  5. 4. 4. 3  and  5. 4.4.4, • 

(3)  Given  (5. 4. 2-6),  constrain  the  AA(kT)  parameters  in  such  a  way 
that  a (kT)  is  reduced  to  a  4x1  vector,  and  then  generate  a 
good  estimate  of  this  vector.  This  approach  is  covered  in 
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Sections  5, 4. 4. 2  and  5. 4. 4. 3. 

(5ie  feature  is  cauiaou  to  all  three  of  these  approaches,  namely 
that  each  of  these  solutions  may  be  formulated  as  a  linear  least-squares 
estimation  problem  leading  to  the  coB5>utation  of  a  matrix  pseudoin^rse . 
^^a^afote  a  summary  of  least— sguares  estimation  principles  is  given  in 
Appendix  (3) /  and  an  overview  of  matrix  pseudoinverse  formulation  tech¬ 
niques  is  given  in  Appendix  C4) , 

Gxe  further  point  should  be  noted.  From  (5. 3.1-2)  and  (5. 4. 2-1)  it 
is  obvious  that  the  parameters  of  ^  do  not  correspond  directly  to  the 
affine  transform  parameters,  but  rather  must  be  adjusted  due  to  the  fact 
that 


This  presents  no  problem,  however  since  we  may  simply  adjust  the  auj  es¬ 
timates  by  a  known  amount,  according  to  (5,4, 3-2), 

5. 4. 4.1  Direct  Solution  of  the  !^ormai.  Equations 
Looking  at  (5. 4. 2-6),  it  is  clear  that  one  straightforward  approach 
is  to  form  a  Markov  estimate  of  a(kT) ,  by  forming  the  pseudoinverse  of  D, 
based  on  a  Q  norm  (for  min  inn  an  vzuriance)  ,  as  follows  s 

a(kT)  -  (DTQD)-lDTQd[(k+l)T] 


(5. 4. 4. 1-1) 

aOcT)  -  DQ^[(k+l)T] 

The  most  obvious  objection  to  this  approach  is  that  the  calculation  of 
Cd'^QD)”^  requires  the  inversion  of  a  6x6  matrix  (or  equivalently  the  so¬ 
lution  of  the  six  corresponding  normal  equations) ,  This  task  is  not  dif¬ 
ficult  using  a  digital  minicomputer ,  but  may  be  formidable  if  restricted 
to  a  dedicated  CCD  discrete  analog  processor,  as  explained  in  Appendix  4. 

5, 4, 4, 2  Modified  Direct  Approach  via  Affine  Transform  Constraints 

The  previous  section  indicated  that  the  general  solution  for  a, 
where  A  was  unconstrained  (except  in  the  sense  that  the  Taylor  Series  ap¬ 
proximation  must  hold) ,  required  the  inversion  of  a  6x6  matrix  (or  the 
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(5. 4. 4. 2-3) 
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It  is  useful  to  define  a  new  vector,  the  constrau.ned  homogeneous  affine 
parameter  vector,  a^,  as 


Cl 

<=2 


(5. 4. 4. 2-4) 


Note  that  the  subscript  c  is  used  here  and  in  the  following  to  denote  "com¬ 
pacted"  or  reduced  dimension  vectors  and  matrices  resulting  from  the  pre¬ 
vious  restrictions ,  and  that  the  above  vector  provides  all  the  information 
required  to  compute  target  rotation  and  dilation  constants. 

Letting  the  newly-formed  total  affine  parameter  vector  be  denoted 
by  a ,  where 


(5. 4. 4. 2-5) 


equation  (5. 4. 2 -5)  may  be  reduced  to 

d  =  [P(,  :  G]a  =  D^a  (5. 4. 4. 2-6) 

where  the  i^^  row  of  is  given  by 

Pc(i^h  row)  =  [(Xiifii+Xi2fi2)  (xilfi2~Xi2fil) ]  (5. 4. 4. 2-7) 


Thus ,  Pc  is  an  Nx2  matrix  of  weighted  spatial  derivatives ,  and  now  Dc  is 
an  Nx4  matrix.  The  form  of  P^  is  consideradjly  simplified  if  an  Nx2  ma¬ 
trix  of  sampled  index  values,  X,  is  defined  as: 

X  =  [xi  ;  X2]  =  [x^T]  k  =  1,2,.  .  .N  (5. 4. 4. 2-8) 

Using  the  definition  of  the  Hadamard  product  (c.f.  Rao  [28],  p.  11),^  Pc 

2)  The  Hadamard  product  of  two  Nxl  vectors,  e.g.  e  =  c*d,  where 
c  =  [c j ]  j  =  1,2,.  .  .N 

d  =  [dj  ]  j  =  1,2,.  .  .N 

is  also  a  Nxl  vector,  and  is  given  by 

e  =  [cjdj]  j  =  1,2,.  .  .N 

Therefore  the  Hadamard  product  represents  a  point  by  point 
multiplication  of  the  elements  of  each  vector. 
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may  be  written  as 


Pc  =  S2] 


(5. 4. 4. 2-9) 


where  *  represents  the  Hadamard  product  operation. 
Dc  may  also  be  put  into  this  form: 


Dr?  * 


Xi*fi+Xi*f2  ^l*f2”*2*^l  I  ^1  ?2 


(5.4.4.2-10) 


Thus,  the  problem  is  now:  given  (5. 4. 4. 2-6),  generate  a  "good"  es¬ 
timate  of  a  (i.e.  ac  and  b) .  Again,  the  least  squares  solution  appears 
promising,  and  as  the  next  section  shows,  since  the  problem  of  forming 
now  involves  the  invarsion  of  only  a  4x4  matrix  with  particular  pro¬ 
perties  ,  a  CCD  implementadile  solution  may  be  readily  obtained. 

Having  constrained  A  in  the  above ,  it  is  also  useful  to  consider 
the  calculation  of  the  actual  affine  parameters  =  and  0  from  (S.4.4.2-lc) 
once  ac  has  been  estimated.  Approximating 

sin9  =  9 

cind 

cos6  :  1-02/2  (5.4.4.2-11) 

cmd,  recalling  the  definition  of  a^  from  (5. 4. 4. 2-4),  it  is  seen  that 


C2  = 

Cl  =  cc(l-e2/2)-l  (5.4.4.2-12) 


and  therefore 

Cl  =  ci+1  =  a:(l-92/2)  (5.4.4.2-13) 

An  estimate  of  the  target  rotation  about  its  centroid,  9,  may  then  be  ob¬ 
tained,  since 

C9  “0  =  0(1+0^/?)  =  0+0V2  =  0  (5.4.4.2-14a) 

Zt  "  cc  (1-02-2  )■■ 


or  more  simply 

0  =  ^2/ci 


(5.4.4.2-14b) 


which  obviously  holds  since  0  was  assumed  small.  Using  (5.4.4.2-13)  and 
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(5.4.4.2-14)  , 
from 


the  target  magnification  parameter,  may  then  be  obtained, 

2 


- SI -  :  - Si -  ;  ci(l+  )  (5. 4.4.2- 

(l-e'^/2)  1-(C2/ci)2 


15) 


or 


Cl 


2 

+  "2/ 


2c]^ 


(5.4.4.2-15b) 


Thus,  a  knowledge  of  a^  uniquely  determines  =  and  0. 


5. 4. 4. 3  Implementation  of  the  Modified  Direct  Approach 
Using  Cline's  Theorem 

Recall  the  estimation  problem,  for  suitable  constrained  A,  is,  to 
estimate  a,  given 

d  =  D^a  (5. 4. 4. 3-1) 


where 


Dc  =  [?l*?l+?2*f2 


a  * 


xi*f2"X2*fl 


fi  fi] 


(5.4.4.3-2a) 


We  may  form  the  L.S.  estimate  of  a  assuming  D,,  has  full  column  rank,  as: 

a  =  Dctd  (5. 4. 4. 3-3) 

(or  equivalently  solving  the  four  normal  equations: 

Dc'^d  =  Dc'^Dca  (5.4.4,;t-4) 

for  a,  however,  for  reasons  cited  in  Appendix  4,  the  preferable  CCD  imple- 
mentable  solution  is  to  form  (Dq) *) •  Also  note  that  while  the  remaining 
analysis  in  this  section  emd  the  following  assumes  the  formulation  of  a 
matrix  pseudoinverse  based  upon  an  identity  matrix  norm  for  simplicity, 
this  work  is  extendable  to  cases  where  it  is  desirable  to  formulate  this 
pseudoinverse  based  upon  a  general  matrix  norm,  Q. 

A  rigorous  mathematical  derivation  of  the  conditions  necessary  for 
Dq  to  have  full  column  rank  appears  quite  difficult.  Therefore,  a  sim¬ 
ple  argument  based  upon  intuitive  reasoning  is  presented,  and  it  is  shown 
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that  for  the  class  of  textured  targets  expected,  this  matrix  should  always 
have  full  column  ramk. 

One  way  to  show  that  has  full  column  rank,  i,e.,  that  the  columns 
of  Dj,  are  a  set  of  four  linearly  independent  vectors ,  is  to  compute  the 
Gramian  of  this  set  of  vectors  (c.f.  29  ,  pp.  217-218)  and  then  show  that 

this  function  is  nonzero.  This  approach  has  limited  analytical  utility, 
however,  since  exact  numerical  values  of  the  vectors  con5)risinCT  are 
not  known  a  priori.  (It  should  be  noted,  however,  that  the  Gramian  cal¬ 
culation  was  incorporated  into  the  simulation  programs  described  in  the 
following  chapter.) 

A  more  reasonable  approach  is  to  recall  the  definition  of  vector  lin¬ 
ear  independence,  i.e.  if  Dc  has  full  column  rcuik,  no  column  of  may 
be  expressed  as  a  linear  combination  of  the  remeuining  three.  It  is  rel¬ 
atively  easy  to  reason  that  and  independent,  recalling  from 

(5. 4. 2 -3)  that  the  k^^components  of  each  of  these  vectors  are 


and 


fk2  = 


af (Xk) 
3X2 


(5,4.4. 3-5) 


where  {xj^};  k  =  1,2.  .  .N  is  the  set  of  all  sample  points.  For  any  rea¬ 
sonably  complex  textured  object  (Section  5.2)  it  is  expected  that  verti¬ 
cal  and  horizontal  target  textures  are  independent,  therefore,  from 
(5. 4. 4. 3-5)  it  is  unlikely  that  fj^  and  f2  are  dependent.  In  fact,  in  con¬ 
sidering  targets  of  practical  interest  (e.g.,  aircraft,  ships,  land-based 
vehicles,  etc.)  it  is  likely  that  the  components  of  f]^  and  f2  will  gener¬ 
ally  be  nonzero,  and  also  "rich”  in  variation,  due  to  the  inherent  texture 
present  in  these  targets . 

It  is  now  necessary  to  show  that  the  remaining  two  columns  of 
are  also  linearly  independent  (of  each  other  cuid  fj^  and  f2)  .  Recall  that 
these  columns  are  obtained  by  a  linear  combination  of  the  Hadamard  prod¬ 
uct  vectors  X2*f2>  X2^*f2  and  X2^*f2,  as  shown  in  (5 .4 .4. 3-2a)  .  It 

is  useful  to  consider  the  structure  of  X]^  euid  X2 ,  where  these  vectors  are 
defined  in  (5. 4. 4. 2-8) .  Figure  7  illustrates  the  effect  of  a  raster  scan 
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on  a  3x3  square  sampling  lattice  with  a  unity  spatial  sampling  interval 
and  the  origin  at  the  center  of  this  lattice .  With  this  strategy  xi  and 
}C2  would  be  of  the  form: 


?1  = 


0 

0 

0 

1 

1 

1 


and 


(5. 4. 4. 3-6) 


Based  upon  the  assumed  variational  "richness"  in  f^  and  f2 ,  and,  as  indi¬ 
cated  in  (5.4.4. 3-6)  ,  also  in  xi_  euid  X2,  and  recalling  the  definition  of 
the  Hadcuiiard  product,  it  is  seen  that  the  formation  of  the  first  two  col¬ 
umn  of  Dq,  i.e.,  the  vectors  (X2^*f2^+X2*f2)  and  (X2^*f2-X2*f2^)  ,  should  yield 
resultant  vectors  which  are  also  "rich"  in  variation  and  independent. 

The  above  reasoning,  in  addition  to  showing  the  independence  of  the 
columns  of  also  links  the  intuitive  amd  mathematical  concepts  of  affine 
pauraparauneter  "estimatibility"  or  target  perturbation  "observability"  or 
"detectadaility . "  For  example,  in  numy  saunple  cases  where  has  column 
raink  less  than  4,  a  humain  observer  would  also  be  unaible  to  determine  cer¬ 
tain  target  perturbations.  The  most  obvious  example  is  the  case  wherein 
f 1  =  f 2  *  0 ,  (i.e.,  the  target  has  no  texture ,  or  equivalently ,  the  tar¬ 
get  interior  is  constant  intensity  throughout).  In  this  case  =  [o], 
and  using  only  information  from  the  target  interior,  it  would  be  impossible 
for  a  humam  observer  to  detect  ainy  tairget  perturbation.  Another  somewhat 
more  realistic  exaunple  is  the  case  wherein  the  target  is  circularly  sym¬ 
metric.  In  this  case  the  column  rank  of  D^,  is  less  than  or  equal  to  3, 
amd  clearly,  using  only  target  interior  information,  a  human  observer  could 
not  detect  target  rotation,  due  to  the  symmetry  of  the  target. 
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Finally ,  it  should  be  noted  that  the  simulation  described  in  Chap¬ 
ter  vn empirically  confirmed  the  preceding  reasoning.  Even  in  cases  of 
very  slightly  textured  teurgets  it  was  found  that  the  Greunian  of  the  four 
vectors  conprising  Dc  was  nonzero,  in  fact,  it  was  never  close  to  zero. 

It  is  felt  that  the  only  practical  case  in  which  the  question  of  column 
rank  could  become  significcuit  is  when  the  target  is  partitioned  into  smaller 
portions  consisting  of  a  small  number  of  textural  samples,  as  in  the  case 
of  Gestalt  Segmentation,  described  in  Section  5. 4. 5. 2.  In  this  case  it 
may  be  possible  for  the  target  to  regionally  lack  texture ,  and  therefore 
TSVIP  estimates  in  these  regions  may  be  unreliable. 

Thus,  if  Dc  has  full  column  rank,  one  way  to  form  Dc"*^  is  to  take 
advantage  of  the  preceding  partitioning  and  apply  Clines  theorem  (Appen¬ 
dix  4)  : 


Dc 


[Pc 


G]^  = 


Pc "^-Pc^gC^-Pc ’’'g  (I -c''‘c)  KG"^  (Pc '^Pc ^  ( I -GC'*';' 
C*+  (I-C*C)KG'’‘(Pc'^)’’Pc'’’(1-GC'^) 


(5. 4. 4. 3-7) 


where 

and 


C  =  (I-PcPc’’')G 

K  -  [I+(I-c'*’c)g'^(Pc^)'’^PcG(I-c'^C)  ]“1 


(5.4.4. 3-8) 
(5.4.4. 3-9) 


Due  to  their  apparent  con^slexity,  little  (if  anything)  is  to  be  gained 
from  the  implementation  of  equations  5.4.4. 3-7  through  5.4.4. 3-9  directly. 
However,  as  reasoned  in  the  following,  C  has  full  column  rank,  so  that 


C'^  »  (c'^O'^c'^. 
c'^'c  »  (c'^O'^c'Pc  *  I 


(5.4.4. 3-lOa) 
(5.4.4. 3-lOb) 


and  (5.4.4. 3-7)  reduces  to 


[Pc  : 


P(;'^-Pc''’GC‘'‘' 

+ 

c 


(5.4.4.3-11) 


Thus,  the  task  of  estimating  a  may  Ise  accomplished  via  a  CCD-imple- 
mentable  processor,  which  only  uses  inner  product  type  operations  (amd 
the  inversion  of  2x2  matrices,  which  is  easily  accon^lished  in  closed  form). 
Recall  (5. 4. 4. 3-8)  and  the  fact  that  Dc  has  full  column  rank.  Then,  from 
(5. 4. 4. 2-9)  and  (5.4.4.2-10)  Pc  has  full  column  rank,  as  does  G,  and 
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(PcTPc)-V 


(5.4.4.3-12) 


C  =  (I-Pc(Pc(Pc'’^Pc)~^Pc'^5G  (5.4.4.3-13) 

Now 

C'^C  =  G^(I-Pc(Pc'^Pc)“^Pc'^)'^(I'Pc(Pc'^Pc)~^Pc'^)G 

=  GT(I-P5,(Pj,Tp^)-lpcT)  (i-p^(p^Tp^)-lp^Tjg  (5.4.4.3-14) 

=  g'P(I-2Pc(PcPc)“^PcT  +  Pc{Pc'^Pc)"^Pc'^Pc(Pc'^Pc)~^Pc'^)G 

=  GT(I-P<,(PcTpc)“^P^T)G 


or 


cTc  =  g'^G  -  g'^Pc(Pc'^Pc)“^Pc'^G  (5.4.4.3-15) 

and  if  C  has  full  column  rank,  (c'^C)  will  be  an  invertible  2x2  matrix  of 
rank  2  (c.f.  Cline  [30]). 

It  appears  difficult  to  prove  that  the  matrix  described  on  the  left 
hand  side  hand  side  of  (5.4.4.3-15)  has  an  inverse.  An  alternate  and  some¬ 
what  devious  approach  is  to  modify  the  approach  of  [31]  in  forming  the 
inverse  of  partitioned  (square)  matrices.  A  brief  digression  will  show 
this  proof.  Recall 


Dc  *  [Pc  •  G]  (5.4.4.3-16) 

and  that  (and  consequently  P^  and  G)  have  full  column  rank.  Therefore 
(Dc'^Dc)”^  exists  and  is  an  invertable  4x4  matrix. 


Expanding 


(Dc'^Dc) 


P  *Pp  *  P 

rc  '^C  I 


|G'^Pc  t  g'^g 


(5.4.3.3-17) 


or,  for  simplicity, 


(Dc'^Dc) 


=11  1 
_ 1_ 

CC12 

- -j- 

=21  ‘ 

“22 

(5.4.4.3-18) 
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Clearly  and  <*22  due  to  the  fact  that  and  G  each  have  full 

column  rank.  Thus  for  any  linear  set  of  equations  of  the  form 

(Dc'^Dc)§  =  y  (5.4.4.3-19) 


there  exists  a  unique  solution  vector  C. 
Partition  C  and  x  as  follows : 


(5.4.4.3-20) 


(5.4.4.3-21) 


Clearly  and  C2  are  then  also  unique.  Rewriting  (5.4.4.3-19)  using 
(5.4.4.3-18)  using  (5.4.4.3-20)  and  (5.4.4.3-21)  yields 


*11  §1  +  *12  §2  =  Vl 

(5.4.4.3-22) 

“2i!i  +  “22!2  =  X2 

.Multiplying  the  first  equation  by  and  simplifying  yields 

§1  *  “ll'^i^l  ■  *ll~^*12f2  (5.4.4.3-23) 

Substitution  into  the  second  equation  yields 

*21*ll"^^l-  “21*11“^®12^2  +  “22^2  =  72 
or 

(“22~“21*11~^*12J ^2  =  y2~*21*ll“^yi  (5.4.4.3-24) 

Since  C2  is  the  unique  solution  to  this  matrix  equation,  it  is  apparent 
that  the  nullspace  of  the  2x2  matrix  W,  where 

W  =  (*22““21*11~^*12) 

is  empty,  thus,  this  matrix  has  full  rank,  (*2) .  Recalling  the  definitions 
of  the  «'3  in  (5.4.4.3-18),  it  therefore  is  proved  that 

g'^G  -  g’^Pc  (Pc'^Pc)"^Pc'^G  =  c'^C  =  W 


is  nonsingular,  therefore,  from  (5.4.4.3-15),  C  has  full  column  rank. 
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Since  (C'^C)”^  exists,  (5.4.4.3-11)  may  be  used  to  form  .  Thus,  to  es¬ 
timate  a  by  partitioning  as  in  (5. 4. 2-7),  (5.4.4.3-11)  may  be  used, 
i.e.  , 


(5.4.4.3-25) 


or,  in  fully  expanded  form 

"(Pc^’Pc)  “^Pc'^  [I-G  (G^G-cTpc  (PcTpc)  -IPc^G)  (I-Pc  (Pc'^Pc)  "^Pc^G)f 

d 

(G^G-gTpc  (P^TPc)  ■IPc'^G) -1  (I-Pc  (Pc'^Pc)  ■^Pc'^G) 

(5.4.4.3-26) 


For  purposes  of  clarity,  (5.4.4.3-25)  will  be  used  to  show  the  simple  CCD 
implementation.  It  is  possible  and  indeed  advantageous  to  estimate  a  us¬ 
ing  parallel  processing  since,  from  (5. 4. 4. 2-5) 


b 


and  therefore  from  (5.4.4.3-25) 

ac  =  (P<-"'-Pc'^GC'*')d  (5.4.4.3-27) 

and 

b  =  c^d 

Equations  (5.4.4.3-27)  and  (5.4.4.3-28)  eire  hereafter  referred  to  as  the 
TSVIP  algorithm,  and  comprise  the  heart  of  the  video  trac)cing  process. 

If  only  b  is  desired,  the  problem  is  greatly  siit^lified,  since  C 
may  be  formed  using  only  inner  product  operations ,  sums ,  and  the  closed 
form  solution  for  a  2x2  matrix,  which  only  requires  the  capability  to  per- 

A 

form  analog  division.  Notice  that  even  if  only  b  is  desired,  this  esti¬ 
mate  is  obtained  in  (5.4.4.3-28)  by  also  considering  the  effects  of  a^, 
even  though  the  latter  vector  may  not  also  be  estimated.  It  is  difficult 
to  overestimate  the  importance  of  this  result  in  view  of  past  efforts  in 
the  pattern  recognition  area.  This  is  in  direct  contrast  to  the  approach 
ta)ten  in  the  next  section,  where  a^  effects  are  modeled  as  "noise.” 
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At  this  point  a  brief  digression  is  in  order.  The  preceding  ainal- 
ysis ,  particularly  for  the  first-time  reader,  may  appear  somewhat  compli¬ 
cated  and  difficult  to  interpret  in  a  physical  or  intuitive  sense.  There¬ 
fore,  the  interested  reader  is  referred  to  Appendix  6,  which  gives  a  sim¬ 
ple  physical  interpretation  of  the  TSVIP  algorithm  approach  (for  the  1-D 
case)  ,  and  presents  several  detailed  numerical  examples. 

Even  though  the  capability  of  analog  division  exists,  in  order  to 
reduce  implementation  complexity  and  achieve  high  speed  processing  it  may 
be  desiraible  to  eliminate  or  minimize  the  need  for  this  operation  (c.f. 
[32],  p.  279).  As  the  following  analysis  shows,  there  are  ways  to  accom¬ 
plish  this. 

While  it  is  not  the  purpose  of  this  work  to  consider  in  detail  the 
various  potential  types  of  CCD  eurchitectures  capable  of  implementing 
(5.4.4.3-27)  and  (5.4.4.3-28),  aui  example  is  included  to  esteiblish  the 
feasibility  of  such  a  design. 

In  order  to  implement  (5.4.4.3-27)  and  (5.4.4.3-28)  the  CCD  proces¬ 
sor  must  have  some  means  of  manipulating  negative  numbers,  in  particular, 
a  four-quadrant  multiplication  and  an  addition  capability.  While  presently 
no  such  device  exists,  it  is  felt  that  CCD  technology  has  this  potential. 
For  example,  fixed  tap  weights  already  exist  (c.f.  [6],  p.  219),  and  it 
appears  that  by  replacing  these  fixed  taps  with  varicible  conductance  NMOS 
trauisistor  weighting  devices  (c.f.  [5],  p.  37) ,  the  capability  of  both 
positive  and  negative  variable  tap  weights  could  be  obtauined.  In  addition, 
methods  are  already  available  for  injecting  and  detecting  bipolar  signals 
(c.f.  [6],  Ch.  Ill,  Section  C) ,  and  the  combination  of  these  two  technol¬ 
ogies  could  result  in  CCD  devices  with  the  aforementioned  desiraible  capa¬ 
bilities.  Therefore,  the  availability  of  such  devices  in  the  future  is 
assumed. 

Assuming  the  matrix  G  aind  the  vector  d  already  exist  (the  estimation 
of  G  is  discussed  in  Section  5. 4. 6.1),  the  first  problem  addressed  is  the 
formation  of  P^.  Recalling  the  definition  of  from  (5. 4. 4. 2-9),  it  is 

easy  to  see  that  auid  52  easily  formed  via  tapped  CCD  registers, 
where  the  register  contents  are  the  vectors  {1  auid  f2  and  the  tap  weights 
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are  X]_  and  X2 . 

Thus,  in  the  implementation  of  (5.4.4.3-28),  the  first  step  could 
be  the  formation  of  Pq~ .  Defining 

VI  =|Pc'^Pcl  (5.4.4.3-29) 

amd  recalling 

Pc"  =  (Pc'^Pc^’^Pc’^  (5.4.4.3-30) 

one  possible  implementation  is  as  follows : 

Step  1:  Form  (Pc’^P^)  and  v^.  Clearly  Pc'^Pc  is  3  2x2  symmetric  ma¬ 
trix,  which  is  easily  formed  using  inner  product  operations. 


(5.4.4.3-30) 


(Pc^Pc)  — 


ei^?i  ei^P2‘i 


?2^Pl 


(5.4.4.3-31) 


It  follows  that 


(Pc^Pr 


to 

-piV 

_-Pl^P2 

Pi'"pi_ 

(5.4.4. 3-32) 


thus  from  (5.4.4.3-30) 

.  np2‘€2^€l^  -  ^?1^P2)?2^1 

viPc  =  - 

^<Pi^P2^Fi^  ^  ^Pl'^?l>P2  J 


(5.4.4.3-33) 


Equation  (5.4.4.3-33)  describes  the  architecture  of  a  CCD  based  Nx2  "pseu¬ 
doinverter,"  which  is  clearly  feasible  with  the  previously  assumed  CCD 
building  blocJcs  (neglecting  temporarily  the  v  term)  . 

Step  2 :  Form  PcPc^ 

The  formation  of  vj^PcPc^  is  as  follows: 

^  ^P2  ^  Pi  ^Pl  '^P2  ^  P2  ^ 

''iPc^c  “  [51  ;  52]  -  (5.4.4.3-34) 

J  <Pl  '^P2  )  Pi  (Pi  ^Pi )  P2 1 


(5.4.4.3-34) 


It  is  again  obvious  that  the  indicated  matrix  multiplication  is  CCD  achiev¬ 
able,  since  each  element  of  the  product  matrix  is  the  result  of  an  inner 
product  operation. 
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Step  3;  Consider  the  formation  of  (I-P(,P(,  ) 
Defining 


LJ  2__1  1  2__,2_ 


:(p^Tp2)p^T+(p^Tg^)p^T 


(5.4.4.3-35) 


it  is  easy  to  see  that  the  formation  of  M  is  again  a  CCD  achievable  opera¬ 


tion  ,  and 


M  =  -ViP^Pc' 


(5.4.4.3-36) 


If  V]_  is  added  to  each  diagonal  element  of  M  (e.g.  have  this  "bias"  built 
into  the  architecture),  the  result,  designated  ,  will  be: 


M+i  =  Vi(I-PcPc+) 


(5.4.4.3-37) 


Step  4:  Form  NLuG.  This  step  again  only  requires  matrix  multipli¬ 
cation.  The  result  is 


‘H+lG  =  V^(I-P^P^^)G 


(5.4.4.3-38) 


=  viC 

Let  V2  =  !  [RflG]T[w+2_G]  |  . 


(5.4.4.3-39a) 


Step  5 :  Using  the  architecture  of  (5.4.4.3-33),  form 
V2  (M+iG)  =  V2  (vj^C) 

=  V2  (viC'^viO  “^viCT  ( 

=  ^(CTC)-ICT 
VI 

-  (^)C" 


(5.4.4. 3-39b) 


Step  6:  Form 


V2  (M+iG)  d  *  ^  C  'd 


(5.4.4.3-40) 


Thus,  using  primarily  CCD-implementable  inner  product  operations,  a  scaled 

«  Vt 

estimate  of  the  affine  trauislation  vector  b  (with  the  scale  factor,  (^) 
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calculable  from  (5.4.4.3-29)  and  (5 . 4 . 4 . 3-39a))  is  obtained.  As  shown  in 
Section  5.4.7  this  form  is  acceptable,  since  b  will  be  again  scaled  by 
the  overall  control  algorithm  (part  4  of  Figure  2)  before  being  used  for 
camera  mount  controls.  Note  also  that  vi  and  V2  were  considered  separately 
throughout.  This  is  advantageous  for  two  reasons: 

(1)  The  problem  of  analog  division  is  circumvented. 

(2)  Should  Pc  not  have  full  column  rank,  vj^  =  0,  and  a  "flag"  to 
signal  the  impossibility  of  matrix  inversion  is  available. 

In  addition  if  V]^  is  close  to  0 ,  a  problem  with  numerical  sen¬ 
sitivity  in  the  algorithm  may  occur,  and  v^  might  also  be  used 
to  identify  this  situation. 

The  procedure  outlined  above  for  implementation  of  (5.4.4.3-28)  is 
only  provided  as  an  example,  and  no  claims  as  to  uniqueness  or  optimality 
are  made.  The  implementation  of  (5.4.4.3-27)  may  be  shown  in  a  similar 
manner.  For  implementation  using  One  analog  divider,  see  C^iapter  VI. 

At  this  point  it  is  useful  to  consider  the  potential  processing  speed 
of  the  aforementioned  CCD-based  implementation.  Under  the  following  set 
of  assumptions,  i.e., 

(1)  The  system  is  in  "steady-state,"  i.e.,  initialization  of  para¬ 
meters  and  initial  segmentation  have  been  accomplished; 

(2)  The  hardware  necessary  to  enable  sampling  of  all  desired  scene 
pixels  simultaneously  is  available ; 

(3)  Information  necessary  for  segmentation  and  spatial  derivative 
data  are  available  from  the  previous  scene;  and 

(4)  A  CCD-implementable  vector  inner  product  operation  may  be  ac¬ 
complished  in  two  or  less  clock  cycles,  once  the  register  con¬ 
tents  and  tap  weights  are  available . 

the  estimates  shown  in  Table  1  may  be  obtained.  As  an  example,  with  a 
1  MHz  system  dork  and  assuming  36  clock  cycles  to  implement  the  TSVIP 
algorithm,  the  total  processing  time  for  this  algorithm  would  be  36  ysec- 
onds . 

The  above  estimates  are  only  intended  to  show  the  potential  speed 
of  the  CCD-implementable  TSVIP  algorithm,  and  should  not  be  taken  as  rigid 
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Operation 

Eguation 

Estimated 
Number  of 
Clock  Cycles 

(1) 

Input  pCi(Ic+l)T) 

- - 

2 

(2) 

Form  d(i,(k+l)T) 

(5. 4. 1-1) 

4 

(3) 

Form  d  (Ic+)  T 

(5. 4. 1^2) 

2 

(4) 

Form  Pq  and  using 

(5. 4. 4, 2-9) 

4 

(5) 

and  ^  from  (2)  and 

G(kT) 

Form  (Pg^Pg) ”1  and 

(5.4.4.3-31) 

3 

(6) 

4. 

Form  vj_Pj,  ' 

(5.4.4.3-29) 

(5.4.4.3-32) 

(5.4.4.3-33) 

3 

(7) 

4* 

Form  v^Pc^c  '  ^+1 

(5.4.4.3-34) 

3 

(8) 

Form  M+iG 

(5.4.4.3-37) 

(5.4.4.3-38) 

2 

(9) 

Form  (^C'*' 

Vi 

(5.4.4.3-39) 

3 

(10) 

X 

Form  (i— )  b 

TL  ’ 

(5.4.4.3-40) 

2 

(11) 

Form  a<; 

**  * 

(5.4.4.3-27) 

4 

(12) 

Scale  b(kT)  for  0(kT) 

(5. 4. 7-2) 

4 

Table 

1 

36  cycles 

Processing  Time  Estimate  for  CCD  In^lementatlon 
of  TSVIP  Algorithm 
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design  specifications.  The  ultimate  tracking  processor  speed  will  be  af¬ 
fected  by  system  clock  frequency,  segmentation  procedures,  amo\ant  of  sys¬ 
tem  pipelining,  filterin  of  TSVIP  estimates,  scene  preprocessing,  scene 
sampling,  exact  algorithm  implementation  (the  preceding  implementation 
is  clearly  inefficient),  camera  mount  dynamic  restrictions,  etc.  The  im¬ 
plementation  of  the  modified  direct  approach  using  Cline's  theorem  by  means 
of  CCD  devices  is  discussed  in  Chapter  VI,  in  which  specific  configurations 
of  formulas  (5.4.4.3-27)  and  (5.4.4.3-28)  are  proposed. 

Finally,  a  note  should  be  made  concerning  scene  dimensions,  i.e., 
the  total  number  of  pixels  processed  at  each  temporal  sampling  instant  in 
an  NXM  scene.  Since  the  TSVIP  algorithm  implementation  is  temporally  ef¬ 
ficient,  but  spatially  complex,  processing  time  should  be  relatively  in¬ 
sensitive  to  scene  dimensions,  provided  assumption  (2)  above  is  valid. 

5. 4. 4. 4  Incomplete  Parameter  Vector  Estimation  Approach 

As  mentioned  in  the  previous  section,  the  estimation  of  the  homoge¬ 
neous  affine  parameters  (^  or  ac)  may  be  of  little  interest,  especially 
in  cases  where  dilation  may  not  occur.  In  this  case  the  method  of  Section 
5.4.4. 3  is  somewhat  simplified,  however,  an  even  simpler  implementation 
results  if  the  effects  of  these  parameters  are  modelled  as  "noise."  As 
shown  below,  however,  the  quality  of  the  estimates  obtained  via  this  ap¬ 
proach  is  questionable. 

Recall  (5. 4. 2-6)  ,  which  may  be  written  using  (5. 4. 2-7)  as 

d  =  Pap  +Gb  (5. 4. 4. 4-1) 

Clearly,  if  ap  =  0  and  b  =  0,  then  d  =  0.  Assuming  the  estimation  of  ap 
is  not  required,  model  the  term  Pap  as  "noise,"  i.e., 

d  =  Gb  +  n  (5. 4. 4. 4-2) 

This  approach  is  similar  to  the  problem  of  system  identification  with  an 
incomplete  parameter  vector  (Eykoff  [33],  p.  199).  Thus,  the  L.S.  esti¬ 
mate  of  6,  using  a  Q  norm,  would  be  obtained  as; 


b  = 

=  (G'^QG)~lG'^Qd 


(5. 4. 4. 4-3) 
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which  is  much  simpler  than  (5.4.4.3-28),  since  only  a  single  2x2  matrix 
inversion  and  a  few  simple  inner  product  operations  are  required.  Using 
(5. 4. 4. 4-1),  however,  it  is  seen  that 


b  =  (g'^QG)  “Ic'^QGb  +  (g'^QG)“^GTqp^ 

b  +  GQ"^ap 


(5. 4. 4. 4-4) 


and  therefore  the  second  term  in  the  above  equation  represents  an  error 
in  the  b  estimate.  In  fact,  unless 


E{GQ+Pap}  =  g 


(5. 4. 4. 4-5) 


this  estimate  will  be  biased,  and  empirical  evidence  (Chapter  VI)  suggests 
there  is  no  reason  to  assume  that  (5. 4. 4. 4-5)  holds.  Furthermore,  in  or¬ 
der  to  obtain  a  "good"  estimate  according  to  criteria  (l)-(3)  in  Appendix 
(3) ,  it  would  be  necessary  to  compute 

Cov{n}  =  E{(Pap)  (P^)"^}^  N  (5. 4. 4. 4-6) 

and  then  choose  Q  =  (minimum  variance)  in  (5. 4. 4. 4-3).  This  is  cer¬ 

tainly  not  easy  to  do  a  priori. 


5.4.5  Segmentation 


As  described  in  Section  2.2,  the  tracJcing  con^uter  must  be  able  to 


separate  the  target  from  the  background,  i.e.,  segment  the  scene  at  each 
temporal  sampling  instant.  For  reliable  TSVIP  estimates.  Section  5.3  in¬ 
dicated  that  only  points  on  the  target  interior  should  be  used.  The  dis¬ 
cussion  of  this  system  requirement  has  been  postponed  until  after  the  de¬ 
velopment  of  the  TSVIP  algorithm,  since  this  algorithm  may  be  used  in  a 
novel  "bootstrap"  fashion  to  aid  segmentation. 

The  automation  of  the  segmentation  capability  inherent  in  the  hviman 
visual  system  is  perhaps  the  most  difficult  tracking  computer  algorithm 
development  task.  As  noticed  by  Rosenfeld  [l];  "It  should  be  emphasized 
that  there  is  no  single  standard  approach  to  segmentation . . . the  perceptual 
process  involved  in  segmentation  of  a  scene  by  the  human  visual  system, 
e.g.  ,  the  Gestalt  laws  of  organization,  are  not  yet  well  understood...." 
Despite  this  difficulty,  numerous  segmentation  approaches  have  been  devel- 
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oped.  Some  of  the  more  popular  approaches  eire  discussed  in  Section  4.4. 

The  varying  degrees  of  success  achievable  with  these  algorithms  is  appar¬ 
ently  a  function  of  the  assumed  picture  model,  and  the  complexity  of  the 
scenes  considered  in  this  work  as  well  as  the  real-time  operational  require¬ 
ment  generally  rule  out  the  use  of  these  simple  approaches. 

In  developing  segmentation  algorithms  for  the  present  system,  a  log¬ 
ical  guideline  is  to  use  a  segmentation  technique  which  is  sviitable  for 
the  type  or  class  of  scene  under  consideration.  Two  general  techniques 
which  appear  promising  are  described  in  the  following.  The  first  uses 
a  statistical  classifier  with  temporal  updating  based  upon  the  TSVIP  al¬ 
gorithm.  The  second  makes  unprecedented  use  of  unsegmented  scene  TSVIP 
estimates  amd  a  Gestalt  law. 

5. 4. 5.1  Statistical  Segmentation  with 
Updating  Using  TSVIP  Estimates 

Before  discussing  these  algorithms  in  detadl,  it  is  useful  to  define 
the  concept  of  a  window,  segmentation,  or  T/B  separation  function. 

As  mentioned  in  Section  5.3.1,  it  is  desired  that  the  sampled  pic¬ 
ture  data  points,  xi;  i  =  1,2.  .  .N,  used  by  the  previously  developed  al¬ 
gorithm,  be  chosen  strictly  from  the  target  interior,  i.e.,  they  represent 
f(xi;kT)  since  X(xi;kT)  =  1.  Since  it  is  only  these  interior  points  which 
are  to  be  considered,  some  guidance  is  needed  in  the  sampling  process, 
and  this  may  be  achieved  using  a  3-D  window  function,  w(i;kT) ,  which  es¬ 
sentially  enables  the  discrete-analog  processor  to  "see"  the  scene  data 
through  an  aperture  or  window,  with  this  window  chosen  such  that  only  tar¬ 
get  interior  points  are  "visible."  Thus,  we  may  define  the  architecture 
of  a  2-D  "scene  sample  enable  register,"  which  holds  the  values  of  w(i;kT) 
for  all  permissible  values  of  i,  and  together  with  control  and  logic  cir¬ 
cuitry,  enable  the  following  decision; 

i  is  an  acceptable  sample  point  if  w(i;kT)  =  1, 
otherwise,  p(i;kT),for  this  value  of  i,  should 
not  be  used  by  the  TSVIP  algorithm. 

Obviously,  if  X(i;kT)  were  known,  ein  obvious  choice  for  w(i;kT)  to  encdsle 
estimation  of  a(kT)  would  be  to  choose 
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w(i;kT)  =  w(i;(k-.l)T)  =  =  X(i;(k+1)T)  =  1 

0  elsewhere 

However,  since  X  is  unknown,  w(i;kT)  must  be  estimated. 

Note  that  the  problem  of  estimating  w(i;kT)  is  analogous  to  the  T/B 
separation  problem,  and  a  nimber  of  well-known  image  processing  segmenta¬ 
tion  approaches  may  be  employed.  For  exainple,  the  missile  tracking  sys¬ 
tem  of  Flachs,  et  al.  [18-24]  assumes  the  target  and  background  pixel  in¬ 
tensities  originate  from  two  different  pdf's,  utilizes  a  "window- frame" 
approach  to  estimate  these  pdf's,  and  then  classifies  on  a  pixel-by-pixel 
basis  using  a  Bayesian  test.  Other  well-known  segmentation  approaches, 
for  example,  texture  edge  detection,  could  also  be  employed. 

An  importcuit  point  should  be  made  concerning  the  linking  of  the  seg¬ 
mentation  process  and  the  TSVIP  algorithm.  It  is  expected  that  conventional 
T/B  separation  will  be  a  relatively  time-consuming  process.  This  is  pri¬ 
marily  due  to  the  fact  that  the  Flachs  approach,  as  well  as  many  of  the 
standeurd  statistical  approaches,  must  re-segment  the  entire  scene  at  each 
sampling  interval ,  thus  the  processing  time  allocated  for  segmentation 
purposes  will  remain  constant. 

The  TSVIP  algorithm  may,  however,  possess  a  clear  advantage  over 
these  conventional  approaches.  Once  the  initial  window  function  (i.e.  , 
w(i;0))  has  been  determined  (perhaps  using  a  conventional  segmentation 
algorithm  or  a  human  operation)  the  TSVIP  estimates  may  be  used  to  update 
this  window,  since  target  translation,  rotation  and  dilation  estimates,  i.e., 
b,  *,  and  0,  will  be  available  at  t  =  kT.  Thus,  using  the  TSVIP  algorithm, 
the  segmentation  process,  in  particular,  the  formation  of  w(i; (k+l)T)  for 
the  scene  at  t  =  (k+l)T  may  be  accomplished  using  the  estimate  of  a(kT). 
Therefore,  as  in  the  derivative  estimation  process,  the  "pipelining"  abil¬ 
ity  of  the  TSVIP  approach  is  apparent. 

The  TSVIP  algorithm  may  therefore  be  conbined  with  the  aforementioned 
segmentation  approach  to  form  the  overall  system  shown  in  Figure  8.  Note 
that  this  processor  contains  the  necessary  algorithms  shown  in  blocks  (1) 
euid  (2)  of  Figure  2,  and  discussed  in  Section  2.2. 
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5. 4. 5. 2  Segmentation  Utilizing  a  Gestalt  Law 

One  of  the  somewhat  sparsely  used  tools  in  scene  segmentation  has 
been  the  use  of  velocity  information  for  segmentation.  Obviously,  this 
information  is  only  useful  when  considering  classes  of  scenes  in  which 
certain  portions  (i.e.,  the  target)  are  moving,  and  clearly  the  image  track¬ 
ing  problem  is  such  a  case.  Potter  [34]  suggests  the  use  of  velocity  as 
a  cue  to  segmentation,  an  idea  also  shared  by  psychologists  [l]  studying 
biological  systems.  Using  the  previously  developed  TSVIP  algorithms,  tar¬ 
get  perturbational  information  (which  includes  velocity  information)  may 
be  employed  in  a  very  straightforward  euid  novel  way  to  enable  scene  seg¬ 
mentation  . 

The  underlying  basis  for  this  segmentation  approach  is  the  Gestalt 
"law  of  common  fate"  {[l],  p.  61)  which  states  that  "...if  a  collection 
of  parts  move  in  unison,  one  tends  to  see  them  as  a  single  figure...." 
Recalling  that  the  TSVIP  algorithm  estimates  a  vector  of  target  perturba¬ 
tion  related  parameters,  a,  it  is  straightforward  to  modify  the  algorithm 
to  employ  this  law.  A  set  of  vectors  {a(q) };  q  =  1,  2,.  .  .B  correspond¬ 
ing  to  (parallel)  application  of  this  algorithm  over  a  set  of  B  non-over¬ 
lapping  2-D  blocks ,  is  coirputed  and  the  blocks  in  which  a  (q)  estimates 
are  identical  (or  close  in  some  vector  norm  sense)  ,  are  classified  as  be- 
loning  to  either  the  target,  the  background,  or  both. 

The  success  of  this  parallel  application  of  the  TSVIP  algorithm  is 
based  upon  the  following  reasoning.  Recalling  (5. 3. 1-5),  i.e.: 

p(x;t2)  =  f  (x;ti)  X(x;t3^)  +  [  '‘^^^•)^X(x;t]^)tf  (x;t]^) 


+  b(x;ti)  [l-X(x;ti)  ]  -  b(x;tx)  ( -  ^  (^-*-b) 


(5. 4. 5. 2-1 


it  is  useful  to  now  examine  the  remaining  two  scene  regions  previously 


mentioned  in  Section  5.3.1,  but  not  considered,  i.e.,  where 


(Background) 


(2)  X  =  0  or  1;  ^  *  "large" 

oX 

For  case  (1),  (5. 4. 5. 2-1)'  in  this  region  reduces  to: 

p(x;t2)  =  b(x;ti) 


(Target  contour) . 


(5. 4. 5. 2-2) 
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=>  p(x;ti) 

therefore  It  is  obvious  that  for  all  x  ia  this  region  (Kx't^)  <■  0,  emd 
if  the  TSVIP  algorithm  were  used,  with  sample  points  from  this  region 
only,  the  estimates  would  yield  a  •  0  or  b  >■  0.  This  is  intuitively  ob~ 
vious,  since  the  background  was  assumed  to  be  slowly  time  varying,  and 
therefore  no  change  is  ejected  over  two  successive  scenes. 

In  considering  case  (2)  ,  a  somewhat  different  ^lppToach  must  be 
taken,  due  to  the  presence  of  the  terms  in  (5.4, 5,2-1)  and  the  binary 

oX 

nature  of  X,  Manipulating  (4. 3. 2-1)',  (4. 3.2-3),  (4. 3.1-2)  and  (5. 3. 1-6), 
it  may  be  shown 

X{x;t2)  f  Kx'tj^) 

that 

dCx'.tj)  -  f (x*,t2)X(x;t2)  -  f {x*,t3_)X(x',tj_)  (5. 4. 5. 2-3) 

+  b(x*,ti)  [X(x*,ti)  -  X(x;t2)] 

Consideration  of  the  two  possible  combinations  of  (X(x;t^),  X(x;t2)) 
values  (i.e.,  (1,0)  or  (0,1))  gives  em  obvious  physical  interpretation  to 
(5. 4. 5. 2-3);  this  is  the  portion  of  the  scene  differance  function  due  to 
either  "covering  \ip"  or  "uncovering"  of  background  points  due  to  target 
perturbations.  Smpirical  evidence  (Chapter  VI)  suggests  that  a  difference 
vector  formed  by  using  these  points  will  tend  to  have  many  coii»onents  of 
relatively  large  magnitudes  (since  there  is  little  spatial  or  temporal 
correlation  between  the  target  and  backgroimd  functions) ,  and  therefore 
if  the  TSVIP  algorithm  is  employed  using  sarnie  points  from  these  re¬ 
gions,  the  a  or  b  vectors  will  tend  to  be  biased  towards  "large”  values. 
Since  these  regions  occur  around  the  target  edges,  it  may  be  desirable  to 
detect  aud  eliminate  samples  origixiating  from  these  regions ,  and  one  pos¬ 
sible  approach  is  to  form  a  histogram  of  the  d  vector  con^xxients  and, 
based  upon  this  data,  only  accept  d^  con?)onents  below  a  certain  thresh¬ 
old.  This  approach  has  been  considered  in  the  simulation  documented  in 
the  following  chapter. 

Summarizing  the  above,  the  following  should  be  noted; 

Cl)  If  a  majority  of  sample  points  are  obtained  from  window  regions 
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interior  to  both  the  Initial  and  perturbed  targets,  the  TSVXP 
algoritha  will  generate  accurate  estimates  of  the  a  or  b  vec¬ 
tors,  which  will  be  close  Cin  a  vector  norm  sense)  for  all 
windows  in  this  region  (an  exception  is  noted  below) . 

(2)  If  a  majority  of  sample  points  eure  obtained  from  window  re¬ 
gions  in  the  background  regions,  the  TSVXP  estimates  of  a  or 

A 

b  will  tend  towards  0. 

(3)  If  a  majority  of  sample  points  are  obtained  from  window  re¬ 
gions  in  the  "covered”  or  "uncovered"  regions,  the  TSVIP  es¬ 
timates  of  a  or  S  will  tend  towards  very  large  or  unreasonable 
values,  which  are  certainly  not  consistent  with  the  original 
assTm^tion  of  small  scene-to-scene  target  perturbations. 

Thus,  the  above  results  may  be  used  in  a  novel  way  to  enable  tar¬ 
get/background  (T/B)  separation,  by  associating  those  window-related  a 
or  b  vectors  vidiich  are  similar  as  representing  regions  vdiich  are  also 
similar,  i.e.,  representing  the  target  interior,  "uncovered* /"covered,"  or 
background  regions.  In  this  way  the  TSVIP  algorithm  may  be  used  perhaps 
as  part  of  a  two-paiss  "bootstrap"  procedure,  wherein  the  first  set  of 
scene  windows  is  chosen  with  dimensions  small  enough  to  generate  a  good 
estimate  of  the  target  contour,  and  then,  based  i^on  these  regional  es¬ 
timates  ,  all  common  or  similar  target  region  data  points  are  then 
"pooled"  for  refined  estimates  of  either  a  or  b. 

Two  final  points  should  be  noted.  First,  in  the  case  where  a  «  0, 
obviously  p(i’,kT)  ■  p(i*(k+l)T),  Thus,  the  preceding  procedxire  would, 
for  every  window,  generate  estimates  closely  approximating  a  «  o.  There- 
fore,  segmentation  would  not  be  possible  in  this  case,  however,  this  is 
of  no  real  concern,  since  if  the  target  is  unchanged,  the  system  does  not 
need  to  take  any  corrective  Mtion.  Second,  since  the  TSVIP  algorithm 
requires  that  the  0^  matrix  have  full  column  rank,  each  window  used  in 
the  Gestalt  segmentation  scheme  must  contain  sufficient  features  such 
that  (Dc^Oj;)  is  nonsingular.  Otherwise,  it  is  not  possible  to  generate 
a  reliable  estimate  of  a  or  b  for  this  window,  and  the  utility  of  the 
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aforemen-tioned  segmentatiion  technique  beccoes  queation£dale , 

Thm  TSVIP  algorithm  may  therefore  be  combined  with  this  type  of 
segmentation  approach  to  yield  the  overall  trzusking  processor  shown  in 
figure  9 .  Note  that  this  processor  contains  the  necessary  algorithms 
shown  in  blocJcs  (1}  and  (2)  of  Figure  2,  and  discussed  in  Section  2.2. 

5.4.6  Consequences  of  the  TSVIP  Algorithm  Approach 
5. 4. 6.1"  Derivative  Estimation 

In  Section  5.4.2, it  was  indicated  that  estimates  of  scene  spatial 
derivatives  were  required  for  the  TSVIP  algorithm.  This  is  not  expected 
to  be  a  critical  requirement,  provided  proper  care  is  exercised  in  ob- 
taining  these  derivatives. 

Perhaps  the  most  obvious  and  straightforward  approach  to  spatial 
derivative  estimation  is  the  use  of  a  simple  difference  approximation. 
This  approach  is  ideally  suited  to  CCS-implementable  calculations .  The 
conditions  necessary  to  insure  the  validity  of  such  an  approximation  axe 
well  known  [35],  [36],  therefore,  for  the  system  considered  here  this 
implies  one  or  more  of  the  following: 

(1)  A  (high  resolution)  sensor  is  available,  with  sufficient  hor- 

(X) 


isontal  and  vertical  sample  resolution,  so  that  — 

used  in  (5.4.2 -3a)  may  be  approximated  as; 


-,  and 


9f(^) 


xj^+A  X2, 

-  f 

L  *2  J  1x2. 


dX2 


(5, 4, 6, 1-1) 


[^1  ri 

l*2+4j  L*2J 


Note  that  this  does  not  imply  that  this  sensor  must  employ  a 
rectangulau:  sampling  lattice ,  but  only  that  at  sampling  points 
the  adjacent  (i.e.,  i-1  and  i+1)  samples  be  sufficiently  close. 


Input  p  (i/JcT) 


To  Ormrall  Control  Algorithm 
for  0(kT)  Calculation 


Figure  9 

Flow  Ch^lrt  for  Video  Trac3cing  Processor  Using  TSVIP  Algorithm 
with  Gestalt  Segmentation 
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Thxs  is  a  consequence  of  the  fact  that  not  al 1  scene  points 
need  be  saa^led,  as  discussed  in  Section  5.4.1, 

(2)  ISie  scene  has  been  sufficiently  low-peiss  filtered  (or  almost 
equi72Llently ,  the  sensor  lens  has  been  sufficiently  defo~ 
cused)  so  that  higher  spatial  frequencies  are  attenuated. 

Thus,  the  spatial  sampling  increment  chosen  will  yield  suffi¬ 
ciently  accurate  derivative  estimates  based  upon  the  differ¬ 
ence  equation  chosen. 

(3)  Random  noise  in  the  scene  has  been  minimized,  perhaps  as  a 
result  of  (2) . 

An  extension  of  the  cUsove  procediire  would  be  the  use  of  more  com¬ 
plicated  difference  functions,  e.g.  repeated  Richardson  extrapolation 
(c.f.  [35],  p.  310).  These  approaches  should  improve  estimate  accuracy 
and  reduce  noise-related  errors ,  while  increasing  spatial  and  temporal 
calculation  costs. 

Other  more  elaborate  derivative  estimation  ^proaches  might  be  the 
use  of  local  polynomial  approximation  (e.g,  "spline"  functions)  or,  the 
use  of  approximate  derivative  reconstruction  functions  (c.f.  Freeman, 
[37],  ch-  5).  These  approaches  may,  however,  significantly  increatse  sys¬ 
tem  complexity  while  yielding  questionable  improvements  in  spatial  deriv¬ 
ative  estimate  accuracy. 

One  final  note  should  be  made  regarding  spatial  derivative  estima¬ 
tion.  As  indicated  in  (5. 4, 2-5)  and  (5. 4. 2-7),  the  estimation  of  all  or 
part  of  a(lcT)  ,  using  a  difference  vector  obtained  by  sampling  scenes  at 
t  >  kT  emd  t  ■  (k-fl}T,  requires  spatial  derivative  estimates  (and  lineu 
combinations  of  the  same)  from  t  -  IcT,  Thus,  the  TSVIP  algorithm  inher¬ 
ently  enable  a  "pipelining"  type  of  processor,  since  the  scene  data  at 
t  ■  JcT  may  be  processed  to  yield  spatial  derivatives  while  simultaneously 
the  scene  at  t  «  (k-^l)T  is  being  sampled  and  the  difference  vector  at 
t  *  (k+DT  is  being  formed.  Thus,  the  raq\iirement  of  spatial  derivative 
estimation  should  not  effect  the  system  temporal  sampling  rate,  but  only 
the  processor  spatial  complexity. 


T 
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5. 4. 6. 2  Validity  of  Approxiaiation — Limits  on  A  and  b 


Tha  major  approximation  ia  the  TSVIP  algorithm  development  con¬ 
cerned  the  modelling  of  target  ten^joral  evolution  using  only  the  first 
order  terms  of  the  exact  2-D  Taylor  series  expansion.  The  validity  of 
this  approximation  is  guaranteed  only  if  it  can  be  shown  that  the  effect 
of  second  order  and  higher  terms  is  negligible.  As  noted  in  Bemark  (1) 
of  Section  5.3.1,  this  implies  that  jj(A-I)x  +  b[[  should  be,  in  some 
sense,  " small , "  since  the  remainder  in  (5. 3. 1-7)  is  known  to  be  (c.f. 
[38],  p.  81) : 


R 


j[Vb]^ 


3f  (?) 
3x2_3xi 

3f  (?) 
3x2 


3f  (?)! 

3x1 35^2  1 


[Ax+b] 


3f  (?) 
3x2  0X2 


(5. 4. 6. 2-1) 


where 

i;e[x,ic+b] 


(5. 4. 6. 2-2) 


Therefore  the  error  introduced  by  the  Taylor  series  approximation, 

A  A 

for  any  value  of  x,  is  proportional  to  | |Ax+bj  p,  weighted  by  a  matrix 
of  spatial  second  derivatives. 

One  straightforward  but  oomputationad.ly  lengthy  approach  might  be 
to  assume  that  the  second  derivatives  of  the  target  textural  ftmction 
are  known,  and  then,  for  each  value  of  x,  use  (5. 4. 6. 2-1)  and  (5. 4. 6. 2-2) 
to  calcTilate  the  worst  case  remainder  as  a  function  of  A  and  b.  Then,  by 
siut2U9ly  constraining  the  remainder  (e.g.  ,  R  -  .01),  establish  limits  on 
A  and  b.  With  some  knowledge  of  the  expected  target  maneuverability 
(«•?•/  Qmjurr  “max'  ^max^ »  ^  then  a  simple  matter  to  set  the  system 

temporal  sampling  rate  such  that  A  and  b  will  be  within  these  limits  at 
all  seunpling  instants. 

A  more  realistic  and  computationally  simpler  approach  might  be  to 
assume  that  the  first  derivatives  of  the  target  textural  function  were  known 
a  priori ,  or  calculable ,  and  that  over  some  multiple  of  the  spatial  sampling 
interval,  i.e.,  k^,  these  paurtial  derivatives  were  very  slowly  vairying. 
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(This  approeuih  was  xised  in  the  simulation  described  in.  Chapter  VI  ^ 
where  it  was  found  that  if  A  was  chosen  to  be  approximately  1/6  of  the 
Nyquist  spatial  sampling  increment,  this  assumption  was  satisfied  for 
k  -  2 , )  Then ,  constraining  target  perturbations  such  that 

||to+b||<kA  (5. 4. 6. 2-3) 

it  is  a  straightforward  procedure  to  use  maximum  target  maneuverability 

•  «  # 

data  Ci.e.  ,  9ma-y »  x^av  aswav)  to  Set  the  temporal  sampling  rate  sxif- 
ficiently  high  in  order  that  t5.4.6.2-«3)  is  satisfied. 

It  is  important  to  note ,  that ,  in  general ,  the  remainder  in 
(5. 4. 6. 2-1)  is  a  fimction  of  x.  For  example,  for  a  simple  rotation  the 
magnitude  of  the  term  ^  increases  with  increasing!  |x|  j.  Therefore,  if 
target  perturbations  were  limited  by  adjustment  of  the  temporal  sampling 
rate  as  above,  it  is  expected  that  a  very  conservative  saapling  rate 
would  be  chosen,  due  to  the  effects  of  large  values  of  | jxj  j  Ce.g,,  at 
the  extremes  or  "corners"  of  the  target).  Furthermore,  since  this  re¬ 
mainder  may  be  thought  of  as  contributing  to  am  additional  error  term  in 
the  formulation  of  the  estimation  equations  in  Sections  5. 4. 4.1  through 
5.4.4-4,  the  effect  of  this  error  on  estimates  of  a  or  b  will  be  inher¬ 
ently  minimized  by  the  least  squares  estimation  algorithms  used.  Thus, 
it  is  more  realistic  to  statistically  consider  the  error  term  due  to 
(5. 4 .6, 2-1)  rather  than  its  maximian  value,  when  choosing  the  temporal 
sampling  rate, 

5. 4. 6. 3  Error  Evolution  and  Minimization 
There  are  a  number  of  ways  in  which  errors  arise  in  the  TSVIP  algo¬ 
rithm  estimates,  emd  no  detailed  analysis  of  all  possible  cases  is  inr* 
tended  here.  Instead,  an  overview  of  these  errors  is  given,  since  many 
have  been  mentioned  in  the  previous  text, 

GenercLlly  errors  in  a  or  b  arise  from  one  or  more  of  the  following 
causes ; 

(1)  Inexact  segmentation  (i.e.,  samples  are  taken  from  points 
other  than  those  on  the  target  interior) .  This  topic  was 
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considered  in  Section  5. 4, 5. 2,  where  it  is  shown  that  this 
situation  may  be  utilized  as  a  novel  aid  to  the  segmentation 
process, 

C2)  Inexact  derivative  estimation.  This  topic  was  mentioned  in 

Section  5, 4, 6,1,  and  it  is  assumed  that  this  effect -is  of  sec¬ 
ondary  importance, 

C3)  Inadequacy  of  ".he  assumed  model.  Extensions  of  the  developed 
model  were  presented  in  Section  4.3,4,  and  it  is  assumed  that 
this  problem  is  not  present. 

(4)  Failure  of  the  Taylor  Series  approximation  in  C5.3,l-7)  to 
hold.  This  problem  C2Ui  only  arise  if  the  system  temporal 
sampling  rate  is  chosen  too  slow,  and  therefore,  if  the 
guidelines  of  the  previous  section  are  followed,  should  not 
occur. 

(5)  Errors  due  to  CCD  implementation  of  the  TSVIP  algorithm. 

This  is  a  very  involved  topic,  highly  dependent  upon  the  exact 
implementation  chosen,  and  ia  suggested  for  future  study. 

(6)  Failure  of  the  target  to  have  sufficient  texture  (i.e., 

does  not  have  full  column  r2Uik) ,  Obviously,  in  this  case 
thete  is  nothing  vrtiich  may  be  done  to  improve  the  TSVTP  esti¬ 
mates.  The  algorithm,  however,  will  output  a  flag  (described 
at  Uie  end  of  Section  5. 4. 4. 3)  indicating  that  the  estimates 
£Lre  unreliable.  Another  somewhat  related  caise  is  the  situa¬ 
tion  where  is  neeurly  singular,  in  which  case  the  poten¬ 

tial  for  severe  algorithm  numerical  sensitivity  is  signifi- 
cwt.  This  problem  may  again  be  eliminated  by  a  careful  a 
priori  investigation  of  the  particular  tracking  application, 

(7)  Random  pictiire  noise.  It  is  likely  that  this  case  will  arise 
in  practice ,  therefore  this  situation  will  be  discxissed 
briefly.  From  (5. 4. 4, 3-3)  this  noise  in  pCi;kT)  will  ob¬ 
viously  affect  a  or  §  in  two  ways: 

(i)  In  the  estimation  of  D^;  and 
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(li)  Through  errors  in  dfCk+DTj, 

Clecurlyi  the  overall  influence  of  both  noise  effects  mentioned  in 
(7)  on  the  TSVI?  estimates  will  be  minimized  if  each  scene  is  pre-pro¬ 
cessed  by  an  appropriate  noise-reduction  filter.  If  such  pre-processing 
is  not  used,  however,  the  TSVIP  algorithm  is  still  expected  to  yield 
good  results  provided  that  a  reasonedsly  noise -insensitive  derivative  es¬ 
timation  algorithm  is  used  Cas  discussed  in  Section  5. 4. 6,1}  and  noise 
statistics  are  approximately  constant  over  a  period  of  several  scenes. 
Wito  these  assumptions ,  the  primary  error  effects  are  due  to  errors  ir: 
d[(k+l}T],  since  from  C5.4.1t1}  and  (5. 4. 1-2) 

d[{k+I)Tl  =«  [p(xi',  (k+l}T  -pCxilkT]]  (5.4.  6.3-1) 

i  =  1,2.  .  .U 
and,  with  additive  noise 

PN(i',kT)  4  pCi;kT)  +  n(i)JcT)  (5.4.6, 3-2) 

where  and  p  represent  the  noise-corrupted  and  the  exact  picture  func¬ 
tions  respectively,  and  n  is  a  3-D  noise  process,  with  assumed  slowly 
temporally  varying  statistics. 

Due  to  the  natinre  of  the  imaging  process  (c.f.  [6],  p.  145),  n  will 
not  be  a  zero-mean  process.  This  is  a  significant  point  often  overlooked 
in  previous  efforts.  However,  from  (5, 4. 6. 3-1)  and  the  assumption  of 
slowly  time  varying  noise  statistics,  it  is  clear  that  the  noise  coiroon- 
ent  of  d{(k+l)Tj  will  be  a  zero  mean  vector,  and  from  i^pendix  3,  the 
TSVIP  estimates  of  a  or  will  then  be  unbiased.  Furthermore,  if  some 
statistical  characterization  of  n  is  possible,  i^pendix  3  aulso  shows  how 
the  least  squares  scheme  Cand  consequently  the  TSVIP  algorithm)  may  be 
adapted  to  obtain  minimum  variance  estimates  for  a  or  b. 

5.4.7  Closed  Loop  Effects  Due  to  TSVIP 

Heretofore,  the  major  emphasis  of  this  work  has  been  on  the  model¬ 
ling  of  scene  evolution  and  the  development  of  the  TSVTP  algorithm  (with 
some  attention  also  having  been  devoted  to  the  segmentation  problem) . 
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Thiis,  those  components  incorporated  In  the  blocics  l2ibelled  "Video  Pro¬ 
cess  I/O  Relationship"  and  "T/B  Sep.  and  Registration"  respectively  in 
Figure  3  have  been  considered.  The  remaining  blocks  in  Figure  3  will 
now  be  addressed,  in  order  to  consider  the  effects  of  the  TSVIP.  algo¬ 
rithm  on  the  overall  closed  loop  system. 

As  mentioned  in  Chapter  2 ,  a  necessary  system  con^nent  is  a  track¬ 
ing  filter,  which  could  improve  TSVIP  estimates  (i.e.,  a(kT))  by  mini¬ 
mizing  the  effect  of  noisy  data,  inexact  T/B  separation,  etc.  Tracking 
filters  of  this  type  are  well  known,  and  an  excellent  summary  of  this 
work  is  found  in  [39]. 

Perhaps  a  more  interesting  problem  is  the  processing  of  the  TSVIP 
estimates ,  a (kT) ,  to  generate  the  actual  camera  gimbal  control  signal 
vector,  0CkT) ,  which  from  (4.1-1)  consists  of  three  components; 


9(kT) 


0a  (kT) 
0e(kT) 
z  (kT) 


(5. 4. 7-1) 


where  the  first  two  components  of  this  vector  2Lre  used  to  adjust  the 
caunera  platform  azimuth  and  elevation,  and  the  last  component  is  useful 
for  zoom  lens  control.  Recalling  from  Section  5. 4. 4. 2  that  the  TSVIP 
algorithm  enables  calculations  at  times  t  »  kT  of  «,  the  taurget  magnifi¬ 
cation  peurameter ,  9 ,  the  target  rotation ,  and  b ,  the  target  horizontal 
and  vertical  perturisations  in  the  sensor  plane ,  the  following  may  be 
reasoned : 

(1)  unless  it  is  desiredsle  to  update  the  segmentation  window  (or 
perhaps  rotate  the  camera  to  keep  the  camera  and  target 
ailigned  in  some  manner]  ,  the  utility  of  the  @  estimate  is 
limited; 

C2)  The  estimate  of  «(kT)  may  be  used  directly  to  form  the  z(kT) 
estimate ,  since ,  from  Appendix  I ,  «  represents  the  amount  of 
target  dilation,  i.e.,  if  the  target  remains  the  same  size 
«  ■  1.0;  and 

(3)  The  estimates  of  b(kT) ,  together  with  a  knowledge  of  the 
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camera  lens  to  sensor  plane  distance,  f,  may  be  used  to  form 
the  control  signals  0^ (kT)  and  Sg (kT) .  Since  these  control 
signals  are  horizontally  and  vertically  separable ,  Figure  8 
illustrates  the  1-D  case.  For  the  target  perturbation  shown 
in  the  figure,  re-centering  the  perturbated  target  at  t  =  (k+l)T 
requires  rotation  of  the  camera  mount  through  an  angle  0p (kT) 
(where  0p(kT)  =  0a  (kT)  if  bj_  =  and  0p  (kT)  =  0e  (kT)  if  bi  = 
b2  in  Figure  8) .  Simple  trigonometry  shows  that  0p (kT)  may 
be  obtained  as : 


0p (kT) 


(5. 4. 7. 2) 


Since  the  quantity  ^  is  typically  very  small.  Thus,  the  gen¬ 
eration  of  the  actual  control  signals,  in  particular,  the  Ceim- 
era  azimuth  and  elevation  controls  is  straightforward. 

Another  important  closed-loop  consideration  is  the  effect  of  the 
camera  controls.  As  mentioned  in  Section  4.3.2,  the  actual  A  and  b  pa¬ 
rameters  which  control  scene  temporal  evolution  are  functions  of  both 
the  camera  mount  controls  and  the  target  trajectory  or  perturbation. 

Since  the  TSVIP  algorithm  developed  herein  operates  on  a  scene-to-scene 
basis,  it  is  necessary  to  adjust  the  estimates  from  this  algorithm  for 
known  camera  movement,  and  this  may  be  accomplished  in  a  very  straight¬ 
forward  manner  by  solving  (5. 4. 7-2)  for  bj^ ,  and  using  this  information 
to  adjust  the  actual  camera  mount  control  signals.  Unless  this  step  is 
taken,  it  is  not  difficult  to  envision  a  system  wherein  a  simple  step 
input  in  target  position  initiates  a  stable  limit  cycle  in  the  closed 
loop  system. 

Finally  it  is  useful  to  consider  the  overall  system  performance 
over  a  large  number  of  successive  scenes  and  various  target  perturba¬ 
tions  ,  perhaps  to  establish  long-term  stability  information ,  or  deter¬ 
mine  the  propagation  of  errors  due  to  initially  inexact  segmentation , 
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lack  of  a  cen'tered  target  at  t  >  0,  etc.  It  is  felt  that  attes^ts  at 
obtaining  this  information  via  walytical  approaches,  in  light  of  the 
mathematical  con^lexities  of  the  target  model  and  the  TSVIP  and  segmen¬ 
tation  algorithms ,  ara  somewhat  hopeless .  Useful  empirical  information 
may  be  obtained  in  the  future  by  modification  of  the  system  simulation 
capabilities  described  in  the  following  chapter 

5 ,5  Summary  of  TSVIP  Properties 

The  primary  advantages  and  disadvantages  of  the  previously  pro¬ 
posed  TSVTP  algorithm  are  simnnarized  below. 

Advantages 

(1)  The  TSVIP  approach  offers  a  mathematically  tractable  model 
and  converts  a  very  nebulous  3-D  signal  processing  problem 
into  one  of  1-D  time-varying  parameter  estimation. 

(2 )  The  TSVIP  algorithms  aire  generally  CCD  implementable ,  euad 
temporcdJ.y  efficient  since  they  inherently  enable  processor 
"pipelining"  techniques. 

(3)  The  TSVIP  algorithm  yields  cm  estimate  of  target  translation 
with  resolution  greater  than  that  of  the  spatial  sampling 
lattice  (i.e.,  no  "moving"  or  perturbed  template  is  neces¬ 
sary)  , 

(4)  The  TSVIP  algorithm  offers  tracking  problem  dimensionality 
reduction  with  a  logical  basis,  i.e.,  in  Section  5.4.1  it  is 
shown  that  only  a  good  statistical  sample  of  target  points 
need  be  chosen,  thus  ad-hoc  procedures  such  as  averaging, 
signatures,  etc.,  are  not  employed. 

(5)  The  TSVIP  approach  does  not  require  the  design  of  algorithms 
insensitive  to  rotation  and/or  dilation,  in  fact  (especially 
in  the  case  of  dilation) ,  it  may  be  desirable  to  estimate 
these  perturbations,  and  this  capability  is  inherent  in  the 
TSVIP  model  formulation. 

(6)  The  TSVIP  ailgorithm  may  readily  be  coupled  with  a  segmenta¬ 
tion  algorithn.to  form  the  basic  video  tracking  processor. 
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In  addition,  the  TSVIP  algorithm  enables  two  novel  segmenta¬ 
tion  approaches,  namely;  (i}  segmentation  window  updating, 
ud  (ii)  Gestalt  ccmmon  fate  segmentation . 

Pi sadvantages 

(1)  The  TSVIP  approach  requires  the  estimation  of  spatial  deriva¬ 
tives.  This  may  lead  to  spatial  oversampling,  however,  since 
it  is  desirable  to  trade  conplexity  in  space  for  tanporal 
speed,  this  may  be  viewed  as  a  system  hardwaure  concern. 

(2)  The  object  to  be  tracked  must  have  adequate  textural  fea¬ 
tures,  otherwise  the  TSVIP  algorithm  estimates  aure  not  reli¬ 
able.  Non-textured  targets  are,  however,  a  trivial  case  and 
woi^d  logically  be  handled  using  simpler  algorithms. 

(3)  Target  perturbations  must  be  restricted  temporally  in  order 
to  insure  the  validity  of  the  first  order  Taylor  Series  ap¬ 
proximation.  However,  this  is  merely  dependent  upon  adequate 
a  priori  choice  of  the  system  temporal  sampling  rate. 
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CHAPTER  VI 

IMPLEMENTATION  OF  THE  MODIFIED  DIRECT  APPROACH 
USING  CLINE'S  METHOD 


In  this  chapter  specific  architecture  is  suggested  for  implementation 
of  the  constrained  TSVIP  algorithm  using  CCD  devices.  The  method  developed 
in  Section  5, 4. 4. 3  will  be  followed.  Recent  developments  of  CCD  program¬ 
mable  transversal  filters  look  promising  for  the  calculation  of  the  inner 
product  of  two  variable  vectors .  This  is  an  essential  function  in  the  pro¬ 
posed  system. 

6.1  Summary  of  Necessary  Formulas 
Recall  (formula  (5. 4. 4. 3-1)  that 

d  =  D^a  (6.1.1) 

where  d  is  the  Nxl  scene  difference  vector,  which  corresponds  to  the  charge 
(proportional  to  light  intensity)  in  the  sensor  sampling  pixels;  Dc  is 
the  constrained  Nx4  matrix  formed  by  the  Nx2  Pc  matrix  and  the  Nx2  G  ma¬ 
trix.  It  is  written 


Dc  =  [Pc  :  G] 

Vector  a  is  the  4x1  total  affine  parauneter  vector. 


(6.1.2) 

The  estimate  of  a  is 


where 


a 


Dcd 


[Pc  i 

r  t  t  t 

Pc  -PcGC 


(6.1.3) 


(6.1.4) 


where  denotes  pseudo  inverse,  and 

C  =  [l-Pc(Pc'^Pc)“^Pc^]G  (6.1.5) 

=  [l-PcPc^]G 


Recall,  also,  that. 


c'^  =  (c'^c)"^c'^ 
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(6.1.6) 

and  P(,~  is  defined  in  a  similar  manner.  According  to  (6.1.3)  amd  (6.1.4), 
the  TSVIP  algorithm  can  be  expressed  as 

^  =  (Pc'*'-Pc'*'GC'^)d  (6.1.7) 

b  =  C~d  (6.1.8) 

Notice  that  once  b  is  estimated,  (6.1.7)  can  be  written  as 

ac  ~  Pq  ^“"^c  Gb 

providing  an  alternate  implementation  for  a. 

6.2  Estimation  of  the  Total  Affine  Parameter  Vector  a 
6.2.1  Estimation  of  the  Translational  Vector  b 
Assuming  that  all  the  sampling  points  are  in  the  target ,  p  =  s  =  f 
is  measured.  The  location  of  the  sampling  points  with  respect  to  the  samp¬ 
ling  lattice  origin  is  )cnown,  and  we  have 


xii  =  value  of  xj^  at  sampling  point  i 
Xi2  =  value  of  X2  at  sampling  point  i 

t 

fil  =  spatial  derivative  with  respect  to  xj^  at  sampling  point  i 

I 

fi2  =  spatial  derivative  with  respect  to  X2  at  sampling  point  i. 


We  can  then  write  expressions  for  Pq  G  as  follows ; 


Pc  = 


G  = 


Pll 

P12 

P21 

P22 

>1 

Pn2 

:  Pi2 

.,2,.  . 

.  ,  N 

fil 

^12 

f21 

f22 

4l 

1 

fN2. 

=  [?1  :  ?2] 


=  [?ii  :  fi2] 


(6. 2. 1.1) 
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It  is  necessary  to  have  an  estimate  of  the  spatial  derivatives  fn 

t 

and  fi2*  The  simplest,  although  perhaps  not  the  most  efficient  way  of 
obtaining  these  estimates  is  to  measure  the  texture  function  (proportional 
to  sampling  cell  charge)  at  the  sampling  points  and  at  two  very  close  points 
in  the  x]  and  x2  directions ,  separated  from  each  sampling  point  by  a  known 
distance  A.  This  is  illustrated  for  a  3x3  (nine  point)  sampling  lattice 
in  Figure  11.  Notice  that  this  seimpling  lattice  correspords  to  the  vec¬ 
tors  X]^  and  X2  given  by  (5.4.4. 3-6)  and  Figure  7.  Recall  also  that  the 
vectors  Xj^  end  X2  are  defined  in  (5. 4. 4. 2-8)  as  X  =  [xi  :  X2  ]  =  xj^  ,  K= 

1,2,.  .  .,N,  and  must  not  be  confused  with  the  vector 


A 


^il 

^i2 


as  defined  in  (4. 3. 3-2). 
For  A  sufficiently  small , 


"af  (x)l  A  ; 

_  3X1 


Xii+A" 

xil' 

-fi 

T 


"3f(X)'!  A. 
3.2 


Xil 

Xil' 

'-"i 

-f 

A 


(6. 2. 1.2) 


for  i  =  1,2 , .  .  .  ,N. 

The  sampling  points  vector  x^  and  A  are  kept  constant, 
constant  and 


Hence 


1 

A 


fii  = 


f  12  ^  £i2^i 


(6. 2. 1.3) 


where  and  fi2+^  indicate  displacement  in  the  x^^  and  X2  directions , 

respectively.  N  measurements  of  fj^,  and  f^2+A  the  value  of 

A  will  allow  us  to  determine,  approximately,  the  matrix  G  as  indicated  in 

figure  12.  Notice  that  the  f's  represent  charges,  and  that  the  i's  rep- 

A 

resent  fixed  tap  weights.  The  implementation  of  negative  coefficients 


Measurement  Measurement  ^il+A 
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Column 
of  G  : 


NOTE:  The  S3niibols  used 
in  this  and  subsequent 
figures  are  defined  in 
Appendix  7. 


Column  2 
c  ~2 


Figure  12  Implementation  of  the  Spatial  Derivative 
Matrix  G 
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(negative  tap  weights)  is  described  in  [47].  Also,  as  mentioned  in  Sec 
tion  5. 4.4. 3,  all  desired  scene  pixels  must  be  sampled  simultaneously  i 


order  to  get  simultaneous  measurements  of  the  f  ' s ,  the  fii+A'®  the/ 

fi2+A'^- 

Once  an  estimation  of  G  is  obtained,  the  matrix  can  be  implemented 
by  means  of  PI/SO  and  SI/PO  registers  with  fixed  tap  weights ,  as  indicated 
in  Figure  13.  Note  that  depending  on  the  specific  characteristics  of  the 

f  t 

devices  used  to  implement  G  =  [fii  Pl/SO  registers  of  Fig. 

13  could  be  eliminated. 

T 

Having  determined  P^ ,  P^  is  also  immediately  available,  because 
in  operations  involving  ,  it  is  only  necessary  to  manipulate  the  ele¬ 
ments  of  Pj,  in  the  right  order  to  obtain  the  correct  result.  The  next 
step  is  to  obtain  (P^.'^P^)”^  for  the  determination  of 

D  t  =  Td  ^-1d  T  (6. 2. 1.4) 


Pc’  =  (Pc^Pc)"-^Pc 


We  can  write 


= 


T  T 

P  P  P  P 

'1~1  ~l-2 


pT'p  pTp 

'2-1  -2-2 


(6. 2. 1.5) 


where 


T  r  ^ 

?iSi  =  I  Pil 

i=l 
N 

?1?2  =  I  PilPi2 
i=l 

T  N  T 

P2PI  =  I  Pi2Pil  =•-  ?1?2 

i=l 

T  ^ 

?2?2  =  I  P  2 

i=l 


(6.2.1.5a) 


(6.2.1.5b) 


(6.2.1.5c) 


(6.2. 1.5d) 


It  is  evident  that  the  implementation  of  the  formulas  above  require 
the  multiplication  of  two  variable  vectors.  This  will  also  be  true  for 
other  parts  of  the  system.  The  problem  can  be  solved  in  a  variety  of 


which  can  be  grouped  into  three  categories : 


\ 


MICROCOPY  RESOLUTION  TEST  CHART 

NATIONAL  BUREAU  OF  STANDARDS  1963-A 


output  From  f . .  output 


PL.  I 
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(a)  CCD  progreumnable  transversal  filters.  This  approach  requires 
the  continuous  variation  of  the  tap  weights  in  accordance  with  the  varia¬ 
tion  of  the  elements  of  one  of  the  two  vectors.  Possible  io^lementations 
for  this  appraoch  are  given  in  [48]  and  [49]. 

(b)  A  combination  of  CCD  registers  and  emalog  multipliers.  Four 
quadrant  analog  multipliers  have  been  designed  with  an  area  of  only  250yin 
by  125uin,  [SO],  [51],  which  would  meOce  it  practical  to  integrate  the  ana¬ 
log  multipliers  in  the  CCD  chips. 

(c)  Analog  multipliers  (AM) . 

Approach  (a)  does  not  seem  appropriate  at  this  time  because  it  re¬ 
quires  a  substantial  amount  of  emcilleury  digital  circuitry.  Furthermore, 
the  programmabj.e  transversal  filter  described  in  [49]  uses,  in  fact,  ana¬ 
log  multipliers  at  each  tap.  These  multipliers  are  in^lemented  by  means 
of  an  MOS  transistor  operating  in  the  line2u:  region  of  the  Iqs  versus  V^g 
characteristic,  and  does  not  appear  to  offer  a  high  degree  of  accuracy. 

This  does  not  mean  that  the  technique  could  not  be  improved  imd  used  ef¬ 
fectively  in  the  future. 

The  implementation  using  aned.og  multipliers  reqxiires  more  devices 
than  that  using  a  combination  of  CCD's  and  analog  multipliers,  although 
its  architecture  is  relatively  single.  Furthermore,  because  most  of  the 
processing  is  done  in  parallel,  the  processing  speed  would  be  limited  m<d.nly 
by  the  upper  operating  frequency  of  the  analog  multipliers. 

The  most  promising  approach  using  CCD's  seems  to  be  the  combination 
of  CCD's  and  analog  multipliers.  This  implementation  will  reduce  the  num¬ 
ber  of  components ;  however ,  for  a  given  scene  san^ling  frequency  some  parts 
of  the  system  will  require  a  much  higher  clock  frequency. 

In  most  of  the  steps  that  follow  for  the  impleirentation  of  the  total 
affine  parameter  vector  a^,  block  diagrams  of  subsystems  using  the  approaches 
(b)  and  (c)  will  be  presented. 

In  Figure  14a,  the  CCD-AM  Implementation  of  Pc^P^  is  shown.  Notice 
that  two  PI/SO  registers,  three  AM's  euid  three  treuisversal  filters  with 
unity  tap  weights  are  necessary.  For  the  AM  implementation.  Figure  14b, 

3N  AMs  plus  3  summers  are  required.  Figures  14a  and  14b  also  include  the 
necessary  blocks  for  the  inversion  of  P^'^P^.. 
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In  Figure  14  we  have  defined 
^11  «IL2 

q21  <122 


N  2  N 

I  Pi2  -I  PilPi2 

i»l  i=l 

N  N 

'I  PilPi2  I  Pil 

i»l  i“l 


(6. 2. 1.6) 


where  is  the  determinant  of 

The  four  elements  of  this  matrix  and  the  2N  elements  of  the  P^^  ma¬ 
trix  are  now  available  in  parallel  form  (Figs.  14  euid  13,  respectively). 
The  pseudo  inverse  of  P,,,  P,,''’  -  (Pj,Tp^) -Ip^T  coim)uted.  It  is 


Pc  *  (P^^Pc)’-"?, 


■Pp  1-lp  T  A 
C  Pc^  Pc 


Qll  Qi2  • • • •  Qiu 

IP2I  222  •  •  •  •  22nJ 


(6. 2. 1.7) 


where,  for  the  first  row 

1  “2  ** 

2lj  ■  Tj-fPjll  pL"Pj2l  PilPi2)'  j=l,2,...,N  (6. 2. 1.8) 

^  i»l  i»l 

and  for  the  second  row 


N  N 


1  2 

®2j  *  Pil“PjlI  PilPi2^ 

i*l  i*l 


j-l,2,...,N  (6. 2. 1.9) 


or, 


Qij  *  Pjiqil+Pj2<Il2 

22j  “  Pj2‘522‘*’Pjl‘Il2 


j“l ,2  , . . .  ,N 


(6.2.1.8a) 

(6.2.1.9a) 


The  CCD- AM  iii5>leiiientation  is  shown  in  Figvire  15a.  Two  PI/SO  regis¬ 
ters  ,  two  SI/PO  registers ,  four  AM ' s  and  two  summers  are  required .  The 
qij '3  are  available  in  parallel  form  from  Fig.  14b  and  the  vectors  P^T 
and  ^2^  from  Figure  13.  The  analog  multiplier  implementation,  shown  in 
Fig.  ISb,  requires  4N  AM's  plus  2N  summers. 

At  this  point  it  is  convenient  to  note  that  although  it  may  at  first 
sight  seem  convenient  to  e:q;>and  matrices  such  as  Pc'^^Pg  and  try  then  to  im¬ 
plement  the  TSVIP  in  terms  of  the  elements  of  the  expanded  matrices,  this 
is  not  definite.  A  little  matrix  algebra  shows  that  for  the  first  element 
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of  Pj,  ^  -  (Pc'^Pc)  ■^Pc’’ 

1  ”2  ** 

2ll  *  TT^PllI  Pi2~Pi2l  PilPi2J 
^  i-1  i»l 

according  to  (6. 2. 1.8).  Neglecting  the  determinant  V][  (for  the  moment) 
and  expressing  the  p's  in  terms  of  Xij's  and  fij's: 

N 

Qll  *  (xilfil+Xl2fl2J  tl  (^ilfi2-’^i2filJ^l 

i»l 

I  I  N 

-  (Xiifi2-Xl2fu)  [I  (*ilfii+*i2fi2>  (*ilfil-*i2filJ  1 
1=1 

(6.2.1.10) 


Frcm  this  expression,  2uid  recalling  that  the  x^j's  are  constant  coefficients, 
we  see  that  each  element  would  require  2  (1+N)  multipliers  for  the  AM  imple¬ 
mentation  or  a  total  of  2N(1+N)2  =  4N+4n2  for  the  Pc’*"  matrix.  This  is  cer¬ 
tainly  much  higher  than  the  7N+5  AM's  required  by  the  method  of  equations 
(6. 2. 1.6)  and  (6. 2. 1.7).  A  similar  situation  develops  for  the  CCD- AM  im¬ 
plementation  . 

Having  obtained  P^^,  the  next  step  (formula  6.1.5)  is  to  determine 
P^Pc^  ^  Pn'  the  subindex  N  is  used  because  the  matrix  is  NxN,  Pf^ 

is 


Pn 


<piiQii+pi2Q2i) 

<P21Qi1+P22Q21) 

^Pni2i1'*‘Pn2®21^ 


fPllQl2+Pl2Q22) 

(P2iQi2'^P22222^ 

*Pn1®12'^Pn2®22^ 


{PIIQIN+PI2Q2N) 

(P2iQin'*'P2222n) 

^%1®1N'''Pn2®2N^ 


(6.2.1.11) 


From  (6.2.1.11)  it  is  obvious  that  an  AM  implementation  would  require  2M^ 
AM's  plus  N^  summers.  This  is  an  excessive  number,  and  for  this  particu¬ 
lar  operation  it  seems  logical  to  consider  only  the  CCD-AM  implementation, 
which,  as  shown  in  Fig.  16  requires  only  2N  AM's,  3  PI/SO  euid  (N-1)  SI/PO 
registers  and  N  summers.  A  potential  problem  can  zurise  due  to  the  high 
frequency  required  for  the  SI/SO  registers  (see  Section  6.3). 


0 


5Ti  of  the  NxN  MatrlJC 

c  c 
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From  (6.1.5)  and  the  definition  of  Pjj  given  in  (6.2.1.11),  we  have 

C  =  [l  (6.2.1.12) 

If  the  N  summers  in  Figure  16  invert  the  input,  then  -Pj|  is  obtained, 
instead  of  Pjj.  If  one  unit  is  added  to  the  element  of  each  column 

of  P^j  at  the  output  of  the  column  registers,  then  [ I  -  Pn]  is  obtained. 
Define  this  NxN  matrix  as 


Sll 

Sl2 

....  S 

A 

®21 

®22 

....  S2JJ 

is* 

®Nl 

®N2 

....  Sjjjg 

[i-Pn1 


Then  C  »  S  X  G  is  an  Nx2  matrix  given  by 

N  N  “ 

I  ®li^il  I  ®li^i2 

i*l  i-1 

N  N 

I  ®2i^il  I  ®2i^i2 

C  =  S  X  G  *  i*l  i=l 


•11  ‘=12 


(6.2.1.13) 


=  21  '=22  (6.2.1.13) 


N  N 

\  <i  .  f !,  V  =  .  f !  "^Nl  ‘=N2 

I  ®Ni^il  I  ®Nifi2  L  J 

i»l  i=l 

I—  J 

Again,  each  element  of  the  matrix  requires  the  sum  of  N  products, 
which  mecuis  that  the  AM  in^lementation  would  require  2N2  multipliers. 

The  CCD- AM  implementation  of  C  is  shown  in  Figure  17.  This  is  the 
most  complex  structure  required  until  now  with  a  total  of  N+2  PI/SO  reg¬ 
isters  ,  2N  transversal  filters,  and  2N  an2Llog  multipliers.  Still,  the  to¬ 
tal  number  of  devices  required  is  much  smaller  than  that  required  for  AM 
in^lementation.  As  cui  example  let  N*21  which  requires  23  PI/SO  registers, 
42  TP  and  42  AM's  which  is  a  total  of  107  devices,  while  for  the  all  AM 
implementation,  441  multipliers  (plus  2N^  summers)  would  be  necessary. 

Notice  now  from  (6.1.5)  that  the  pseudo  inverse  matrix  C^  is  computed 
exactly  in  the  same  way  as  P^.  In  other  words,  Figures  14  emd  15  are  appli- 
cad)le  to  obtain  C”^  with  the  difference  that  the  inputs  are  now 
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Figure  17  Implementation  of  the  C  Matrix 
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®11 

®12 

<=21 

A 

C22 

CNl 

?2  = 

<3n2 

instead  of  and  P2  • 

Recall  that  if  all  sampling  points  are  inside  the  target,  p  =  f  =  s 
and  we  have 


d  =  p[(k+l)T]  -  p[kT] 


(6  .2.1.14) 


where  T  is  the  sampling  interval.  The  first  part  of  Fig.  18  shows  the 
implementation  of  d  by  means  of  CCD  registers  and  summers. 

The  matrix  C^  can  be  expressed  as 


Hence 


c{i  CI2  ....  CiN 

+  t  t 

C21  C22  C2N 


N  4. 

I  '^li'^i 
i=l 

N  t 
I  <=2i'^i 

i=l 


(6.2.1.15) 


(6.2.1.16) 


The  CCD-AM  implementation  of  this  expression  is,  once  again,  much 
more  convenient  than  the  one  using  AM's  only.  The  second  part  of  Fig. 

18  shows  the  implementation  of  (6.2.1.16) 

6.2.2  Estimation  of  the  Constrained  Rotational-Dilation  Vector, 

The  expression  for  ac  in  terms  of  Pc*»  G,  C"*"  and  the  scene  differ¬ 
ence  vector  d  can  be  written  in  several  equivalent  forms,  e.g., 

4c  “  (Pc^~Pc"‘^C*)d  (6.1.7a) 

=  Pc^d  -  Pc*Gb  (6.1.7b) 

The  matrices  vectors  d  and  b  have  already  been  imple¬ 

mented,  and  consequently  the  implementation  of  a^.  is  relatively  simple. 
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Figure  18  Implementation  of  the  Scene  Difference  Vector,  d,  and  the  estimate  of 
the  Translational  Vector,  b. 
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It  is  important  to  note  that  no  matrix  inversions  are  required. 
The  best  configuration  is  obtained  if  ac  is  written 


?c  ™  Pc  ^(*5  "* 


(6. 2. 2.1) 


Then  we  have 


A 

Gb  *  h 


filbi+fi2b2 

w 

1  1 
^21^1  ^22^2 

a 

^2 

P2_ 

1  1 

%1^1  ^N2^2 

• 

hN 

(6. 2. 2. 2) 


ac  =  Pc‘^(d-h) 


■'ll 


■'12' 


■'IN 


•(di-h,)- 

ai 

(dj-hj) 

a 

*2 

,{djj-hN)_ 

(6.2.2. 3) 


[221  ^22  22n| 

The  implementation  of  ag  is  shown  in  Fig.  19.  Notice  that  the  com¬ 
ponents  of  Pc^»  qJ  and  Q2  avilable  in  serial  form  from  the  PI/SO  regis¬ 
ters  of  Fig.  15a.  The  saune  is  true  for  the  con^onents  of  G,  fj^  amd  fi  from 
the  PI/SO  registers  of  Fig.  13.  Finally,  the  vector  d  is  avadladale  in  par¬ 
allel  form  from  Fig.  18.  rr  i 

fs 

This  completes  the  implementation  of  a  >  |b  J .  As  can  be  observed 
by  inspection  of  Figures  12  to  19,  the  use  of  CC25's  auid  AM's  makes  it  pos¬ 
sible  to  implement  the  system  without  resorting  to  continuously  programmable 
tramsversal  filters.  The  inclusion  of  AM's  on  chip  will  probably  guaran¬ 
tee  a  higher  degree  of  accuracy  thauri  the  use  of  programmable  transversal 
filters  such  as  those  described  in  [49].  A  block  diagram  of  the  complete 
system  is  given  in  Fig.  20. 


6.3  Sampling  Rate  and  Other  General  Considerations 
6.3.1  Sampling  Rate 

Assuming  that  the  N  samples  at  points  x^,  i  >1,2,...N  plus  the  2N 
s£unples  at  points  ^i+ilX2  total  of  3N  seunples)  are  availedsle  in 

parallel  form,  a  knowledge  of  the  focal  length,  f>  the  focal  point  to  tar¬ 
get  plane  perpendicular  distance,  Z,  and  the  velocity  of  the  target,  v-p. 
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in  the  target  pleuie ,  allows  calculation  of  the  translational  vector  b  (as¬ 
suming  that  dilation  is  not  involved)  for  the  san^ling  interval  T.  This 
cissumes  a  sampling  rate,  fs  samples/second.  With  reference  to  Figure  21 
(for  simplicity  we  have  considered  movement  in  one  dimension) ,  assume 

f  ■  50  mm,  Z  “  100  m,  v^  ■  9000  ^  (mach  7.35) 

Then 

Abi  -  I  Abi-r  *  5xlO-'«Abi^  (6. 3.1.1) 

The  basic  assumption  for  the  main  body  of  this  wor)c  has  been  that  the  bacJc- 
ground  vauries  (evolves)  slowly.  If  for  the  chosen  seunpling  rate  Abj^  is 
very  small ,  then  we  cam  certainly  assume  that 

b(x,ti)  :  b(x,t2) 

Here,  once  again,  recall  that  b  is  the  taurget  tramslational  vector,  and 
b(x,t)  is  the  background  fxinctlon. 

With  the  figures  given  aOsove,  the  speed  in  the  image  sensor  plane 
is 

»  5xl0“^x9xl0^  -  4.5  Km/hr 
or 


-  1250  mm/sec 


This  appears  to  indicate  that  a  sauopling  rate  (for  all  pixels  in  paurallel) 

of  F»1  MMZ  is  adequate.  Then 

C  C  ^  1250 

^ik+1  -  ‘’ik  ^  ^ 

■  bij^  +  1.25  pm 


The  taurget  (in  the  target  plane)  will  move  by  a  distance 


.C  _  9x10^  1 

^^^T  3600  ‘  lo^ 


2.5  mm 


The  elements  of  am  image  sensor  device  are  extremely  small.  For  example 
the  Reticon  MC520  100x100  matrix  camera  has  a  distance  of  60um  between  cen¬ 
ter  of  elements.  The  Fairchild  CCD211  area  image  sensor  has  244x190  elements 


with  dimensions  of  14um  horizontally  and  18  m  vertically.  For  both  casieras, 
the  change  in  b  in  the  isuge  sensor  plane,  >  1.25um,  is  less  than  one 
tenth  the  smallest  dimension  of  the  element ,  t^ich  means  that  all  the  eis- 
suB^tions  and  approximations  will  be  ao^ly  satisfied.  Obviously,  if  z>100 
m  and/or  v^9000  Km/hr,  a  better  situation  will  exist.  Talcing  now  the  oppo¬ 
site  extreme  of  a  very  short  distance,  Z  »  1  m,  and  a  very  low  speed,  v^  ■ 

5  m/sec,  v^g  »  0.25  m/sec,  Ab^  ■  0.2Sub  which  is  even  a  better  estimation 
than  the  other  case. 

6.3.2  Ooamatibility  of  CCD’s,  AM's  and  Assumed  Saat>ling  Rate 

For  the  assumed  sas^ling  rate  of  F>1  MHZ  some  of  the  CCD  devices  and 
AM's  will  have  to  work  at  a  frequency  of  N-F.  Assuzaing  a  10x10  sas^ling 
lattice  (which  once  segmentation  is  achieved  will  be  more  than  enough]^  N  > 

10  and  these  devices  will  have  to  operate  at  10  mz.  With  presently  aved.1- 
able  technology  this  operating  freqiiency  will  represent  no  problem  for  the 
CCD  devices ,  [52 ] .  Integrated  analog  multipliers  are  presently  available 
with  bandwidths  of  only  1  MHZ,  ^a. though  the  availad)ility  of  10  MHZ  hybrid 
multipliers,  [53],  seems  to  indicate  that  IC  AM's  with  a  bandwidth  in  this 
range  will  be  availedzle  in  the  near  future.  Finally,  some  SI/PO  registers 
require  a  clock  frequency  of  N^F,  «Aich  for  N-10,  F«1  MHZ  will  be  too  high 
for  presently  available  CCD  devices  (see  Fig.  16).  However,  peristaltic 
CCD's  with  operating  frequency  of  more  than  135  MHZ  have  been  reported  in 
the  literature  [54]. 

In  conclusion,  a  CCD  implementation  of  the  TSVIP  appears  to  be  both 
possible  and  practical.  For  a  sampling  rate  of  1  MHZ  and  lattice  size  N  « 
10,  some  presently  available  AM's  and  CCD's  may  not  provide  the  necessary 
perf omuuice .  However,  a  prototype  can  be  constructed  with  a  lo%«er  sampling 
rate  F  *  0.5  MHZ  and  n  ■  5,  which  will  result  in  a  highest  operating  fre¬ 
quency  of  12.5  MHZ  for  the  CCD's,  and  of  2.5  MHZ  for  the  AM's.  Alternately, 
the  tracking  velocity  can  be  reduced  to  make  operation  easier  to  achieve. 
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CHAPTER  VII 

SIMULATION 


‘Rie  con^uter  simulation  described  in  this  chapter  was  initiated  in 
order  to  verify  empirically  the  preceding  theoretical  results ,  in  partic¬ 
ular  the  validity  of  the  TSVIP  algorithm.  The  interactive  video  tracking 
simulation  program  developed  is  also  useful  as  am  empirical  research  tool. 
Using  simple  analytic  fiinctions  to  represent  target  texture,  simulated 
TSVIP  algorithm  performance  is  shown  to  follow  theoreticail  predictions  very 
closely. 

7.1  Program  Description 

A  detailed  flow  chart  and  listing  of  interactive  video  tracking  pro¬ 
gram  TSVPD  is  included  in  Appendix  5.  Smooth  analytic  functions  are  used 
to  simulate  the  target  textural  function,  thus  this  fimction  may  be  perturbed 
exactly  via  the  affine  transform,  smoothness  was  necessary  in  order  to 
permit  estimation  of  spatial  derivatives  using  a  sinf>le  difference  approx¬ 
imation.  A  21x21  scene  is  used,  and  background  and  scene  noise  values  are 
taken  from  normal  distributions.  The  program  confutes  the  original  and 
perturbed  scenes  in  accordance  with  the  model  of  Section  4.3.3,  amd  then 
uses  scene  differences  and  the  TSVIP  algorithm  (emulating  the  CCD  implemen¬ 
tation  of  Section  5. 4. 4. 3)  to  estimate  taurget  perturbations.  The  program 
itself  is  well-documented  through  the  use  of  extensive  comment  statements 
auid  the  flow  chart  of  App.  5.  However,  for  illustration  purposes,  a  sin^- 
llfied  flow  chart  of  program  operation  is  shown  in  Figure  22 .  There  are  a 
total  of  25  paurameters  or  options  which  must  be  input,  and  these  are  described 
in  the  program  listing. 

7.2  Simxilation  Summary 

As  mentioned,  simulation  was  undertaken  to  verify  theoretical  deri¬ 
vations  ,  particularly  the  validity  of  the  Taylor  series  approximation 
and  estimation  algorithm  inqplementation  using  Cline's  theorem.  There¬ 
fore,  no  extensive  parameter  or  sensitivity  analysis  is  presented.  A 


Qioomm  Sp«tl«l  Saapliag  Xataxvml,  Target  Bouadarias, 
BacJcgroaid  Statistics  and  Scans  Haadon  Noisa  Statistics, 
roca  Initial  Scans,  p(4/0)*  _ 


Qioosa  Affina  Parturbation  Parasatars  A(0)  and  hiO) . 

Pen  Partxssbad  Scans,  p(4>7},  and  than  Diffarenca  Scans, 
d(i',T).  _ 


_ t _ 

Oioosa  Singla  Window  Function  os  Gastalt  Sagnantation 
Option. 


_ ± _ 

Qioosa  Saapling  Stratagy,  i.a.,  Z.  Foam  oiffaranca 
Vactor,  ^(T) . 


_ t _ 

Estisata  Spatial  Oarivativas  from  pC^^O),  i.a.,  G(0) . 

Form  0  (0) . 
c 


_ ± _ 

Apply  CCD'^Xmplaaantabla  Varsion  of  TSVXP  Algorithm 
asing  Clina's  Thaorsm.  (aq.  5.4.4.3-25).  Canputa  a(0). 


Oompota  0  and  a  from  Compmrm  Exact  and  Bstimatad 
Paramatar  Valuas  and  Canputa  Errors. 


Figora  22 

Flow  Chart  of  Intaractiva  Vidao  Traclcing  Simulation 
Program  TSVPD 
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ntabsr  of  topical  sa^la  casoa  may  b*  iiMd,  howowr^  to  conflza  th« 

th«or7. 

nwi«M  otharviM  notad,  tha  foUoiilng  data  was  usad  to  obtain  tha 
graphical  data  shown t 

(1)  Targat  Taxttmd.  Function— 


ttiw  ■  [Sifjiil  ^  0.314] 


(7.2-1) 


(2)  Spatial  Sampling  IntarTal— 


A  “  0.2  "sensor  units'* 

(Ipproximataly  lOX 
the  Nyquist  sampling 
rata) 

(3) 

Background  Statistics— 

b(i?0)  -v  M(4,l) 

(4) 

Targat  Outline- 

Sqoara;  16izl6A  (Horizontal  Z  vaztical 

diaansions) 

An  important  slmulatian  rasult  is  a  conparison  of  axact  and  asti- 
aatad  targat  translational  paraaiatars,  i.a.,  b  jmd  b.  Figure  10  cont- 
paras  actual  translation  paramatars  and  TSVIP  algoritfaa  astimatas  for 
tha  casa  of  «  >  1.0  and  6  •  0.0  (i.a.,  tha  targat  doas  not  ajqparianca 
rotation  or  magnification) .  Two  points  should  ba  notadi 

(1)  nia  sagnantation  window  was  manually  choaan  such  that  only 
targat  intarior  points  ware  saaplad;  and 

(2)  For  simplicity  only  tha  first  conponants  of  tha  axact  and  as- 
tlaatad  b  vaetora  ara  shown ,  howawar ,  tha  actual  axact*  par- 

m 

turbations  wara  aqoal  la  tha  horizontal  and  vartical  dlrac- 
ticns,  i.a.,  bi  *  b2f  and  tha  rasults  for  bi  and  b2  ara  simi¬ 
lar. 

As  shown  in  Figure  23,  tha  TSVIP  algorithm  parfozaanca  is  guita  good  for 
|bi|<0.5.  Hota  that  as  tha  targat  partiirbations  increasa  in  oagaituda 
tha  astimatas  vorsan,  as  pradictad  in  Saction  5.4. 6. 2.  It  should  also 
ba  noted  that  this  aopirlcally-obtainad  I/O  function  is  approximataly 


Figure  23 

TSVIP  Algorithm  Translational  I/O  Function  (O'O.O;  a«1.0) 

(Other  Parameters  in  Text) 
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linear /  a  very  useful  fact  for  closed-rloop  stability  analysis,  Obvi- 
oosly  (in  this  figure  and  the  following) ,  the  dotted  line  shown  with  a 
slope  of  4>1  represents  the  ideal  or  erroT'v^ree  traclcing  processor  I/O 
function. 

Perhaps  an  even  more  interesting  case  is  shown  in  Figure  24,  vrtiere 
«  a  .97  and  9*3*^  (i.e.,  in  addition  to  translation,  the  target  expe¬ 
riences  rotation  and  dilation) .  The  TSVIP  translational  estimate  accu¬ 
racy  in  this  case  is  virtually  unchanged  from  that  of  Figure  10,  in 
other  words,  the  TSVIP  algorithm  is  empirically  shown  to  be  insensitive 
to  rotation  and  dilation.  In  fact,  this  es^irical  resTilt  has  been  veri¬ 
fied  (for  this  sample  function  u  well  as  severzil  others)  for  cases 
wherein  |«-l|  ■■  .1  and  |d|<10^.  These  empirical  results  therefore  con¬ 
firm  perhaps  the  most  ir^ortant  tracJcing  eilgorithm  design  goal  mentioned 
in  Section  2.1,  i.e.,  insensitivity  to  rotation  and  dilation. 

dearly,  the  TSVIP  algorithm  based  system  is  a  MIMO  system,  since 
there  are  four  scalar  I/O  functions,  i.e., 

(1)  bi  vs.  bi; 

(2)  62 

(3)  9  vs,  9;  and 

(4)  «  vs,  «. 

The  empirical  results  for  all  four  I/O  functions  were  fotmd  to  be  simi¬ 
lar,  e.g. ,  Figure  25  shows  9  vs.  9  for  the  case  of  b]^  b2  *  0.1  and 
«  -  1.0. 

An  obvious  practical  consideration  is  the  effect  of  random  scene 
noise  (as  distinguished  from  inexact  segmentation  which  causes  back¬ 
ground  data  to  be  processed  and  results  in  a  different  type  of  process 
"noise”)  on  the  TSVIP  algorithm  estimates.  A  sample  of  this  type  of  em¬ 
pirical  result  is  shown  in  Figure  24.  In  order  to  simulate  the  type  of 
noise  process  described  in  Section  5. 4. 6. 3,  a  zero  mean,  normally  dis¬ 
tributed,  independent  random  process  was  generated  and  variates  with 
negative  values  were  discarded.  Thus,  additive  scene  noise  was  there¬ 
fore  always  positive,  originating  from  a  truncated  normal  distribution. 
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As  shovm  in  Figure  26,  this  type  of  additive  noise  affects  the  TSVIP  eti- 
gorithm  estimates  considerably.  Prom  Sections  5. 4. 6.1  and  5. 4, 6. 3 
is  to  be  ei^ected  for  two  re2isons: 

(1)  The  TSVIP  algorithm  requires  the  estimation  of  spatial  deri¬ 
vatives.  niis  was  done  in  program  TSVPD  using  a  sia^le  dif¬ 
ference  equation  (i.e..  Equation  5 ,4.6.1-!.l)  ,  which  is  known 
to  be  a  noise-sensitive  approach. 

(2)  The  TSVIP  algorithm,  due  to  the  inherent  model  approximation, 
operates  on  small  scene-to-scene  pixel  intensity  differences. 
“Oierefore,  in  noisy  scenes  the  problem  of  numerical  cancella¬ 
tion  (c.f.  [35],  p.  28)  may  occur. 

In  order  to  determine  en^iriceO-ly  the  overall  contribution  due  to  each 
of  the  above  error  sources.  Figure  24  also  shows  the  TSVIP  algorithm  es¬ 
timates  obtained  by  using  error-free  spatial  derivative  data.  Clearly, 
the  estimates  are  significantly  improved,  and  therefore  the  comments 
made  in  Sections  5. 4. 6.1  and  5. 4. 6. 3  are  verified. 

It  is  also  useful  to  consider  the  alternative  version  of  the 
tracking  processor  described  in  Section  5. 4, 4. 4.  This  approach  (Equa¬ 
tion  5. 4. 4. 4-3)  yields  a  somewhat  simplified  algorithm  which  only  esti¬ 
mates  translational  parameters  and  models  target  rotation  and  dilation 
effects  as  noise  in  the  estimation  process.  As  mentioned  in  Section 
5. 4. 4. 4,  however,  this  type  of  model  may  yield  biased  estimates.  This 
reasoning  is  enpirically  confirmed  in  Figure  27,  where  target  dilation 
and  positive  or  negative  rotation  are  shown  to  cause  a  bias  in  the  algo¬ 
rithm  translational  estimates.  As  predicted  by  (5. 4. 4. 4-4),  for  con- 
•■tAot  «  and  0  this  bias  is  also  constant  over  the  range  of  b  considered. 

As  mentioned  in  Section  5, 4. 5. 2,  an  addi tionai  error  arises  in  the 
TSVIP  algorithm  vrtien  the  segmentation  window  is  too  large  and  pixels 
fron  other  them  the  target  interior  eire  sampled.  This  effect  on  TSVIP 
algorithm  translational  estimates  is  shown  in  Figure  28 ,  for  two  cases 
with  differing  background  noise  statistics.  Here  the  segmentation  win¬ 
dow  was  chosen  25%  larger  than  the  ntavimum  permissible  value,  therefore 
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Effect  of  Randoa  Additive  Picture  Noise  on  TSVIP  Algorithm 
(  Noise  Process  Described  in  Text) 
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rigura  27 

Translational  I/O  Function  of  Sioplified  Estimation  Algorithm 
Which  Models  Rotation  and  Dilation  as  Noise 


rlgoxtt  28 

TSVIP  Algorithm  SegnMotatlon  Window  Effect  (8>0.0*  a*I.O) 
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as  snntioiisd  in  Sactiao  S.4.5.2f  difference  values  and  derivatives  from 
"uncovered”  and  "covered,"  and  background  regions  are  processed.  As 
shown  by  Figure  28 ,  algorithm  performance  is  not  good,  ho%*ever,  it  is 
^  note  the  significant  difference  in  accuracy  between  the 
cases  v*ere  b(i;o)  N(0,0)  (i.e.,  all  "white"  background)  and  b(i',0)  -v- 
If(4,l}.  Performance  is  significantly  better  for  the  case  of  a  "\dtite" 
background ,  and  it  is  felt  this  occurs  due  to  the  background  homogeneity 
and  the  sharp  target/background  edge.  In  this  case  the  algorithm  actu¬ 
ally  uses  edge  Infomation  correctly  in  determining  target  perturbations 
(neglecting  the  relatively  insignificant  bleis  due  to  background  pixels) . 
In  the  case  where  b(i;0)  N(4,l)  the  independence  of  the  background 

variates  causes  large  errors  in  derivative  estimation.  Also,  the  rap¬ 
idly  (spatially)  varying  background  causes  difference  scene  pixel  inten¬ 
sities  and  spatial  derivatives  to  lack  correlation,  therefore  (5. 3. 1-7) 
is  not  even  ^proximately  valid  over  thia  region. 

Finally,  an  empirical  exas^le  of  the  Gestalt  segmentation  tech¬ 
nique  is  presented  in  Figure  29.  For  this  case  an  0^X164  target  was 
chosen,  and  Gestalt  segmentation  windows  with  dimensions  of  4AX5A  were 
used.  Despite  the  fact  that  each  window  only  contained  30  points,  good 
results  were  obtained,  as  shown  in  the  figure.  For  this  case  the  exact 
target  pertTirbational  parameters  were: 

«e  ■  1.0 
0  -  0.0 
bi  -  -0.2 
1>2  ■  0*0 

As  shown  in  the  figure,  the  three  types  of  estimates  predicted  in  Sec¬ 
tion  5. 4. 5. 2  are  obtained.  Obviously,  for  windows  wholly  containing 
background  points  the  estimates  indicate  no  measxirable  perturbation. 
Furthermore,  for  windows  encoa^jassing  target  edges,  the  estimates  indi¬ 
cate  either  very  slight  perturbations  (due  to  a  majority  of  background 
pixels  and  lack  of  a  sharp  target  edge)  or  large  perturbations  (due  to 
large  difference  scene  pixel  intensities  and  erroneous  derivative 


Gestalt  Segmentation  Using  TSVip  Algoritlin 
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ttstiaatas  resulting  fro*  the  "covsring"  or  "uacovuring"  of  the  back¬ 
ground)  .  Thasa  aatiaatas  ara  aasily  distingulshabla  from  thosa  obtainad 
using  windows  wholly  containing  target  interior  points.  These  latter 
astiaatas,  as  shown  in  figure  29  ,  ara  in  axcallant  agraamant  with  the 
•xact  target  parturbationad  paumnatars. 

Finally^  since  the  aforeaentioned  simulation  utilized  a  scene  of 
relatively  small  dimensions  (in  real  a^lications  256x256  to  512x512 
pixel  scenes  are  expected)  and  relatively  slightly  textured  target  func¬ 
tions  (due  to  scene  dimension  and  derivative  estimation  constraints) ,  it 
is  expected  that  algorithm  performance  in  actual  applications  may  be 
*ome%»hat  better/  as  a  result  of  a  considerably  leurger  of  pixel 

samples  and  much  greater  variance  in  the  target  texture. 
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CHAPTER  VIII 

CSDNCXUSIONS  AND  SUGGESTIONS  FOR  FUTURE  STUDY 

A  suitable  algorithm  for  tracking  in  realistic  situations  consis¬ 
tent  with  using  small  portable  hardware  has  not  been  found,  so  it  has 
been  necessary  to  do  research  on  algorithm  development.  The  TSVIP  al¬ 
gorithm  presented  herein  is  a  novel,  CCD-implementcdale  (with  the  possi¬ 
ble  exception  of  some  analog  multiplication)  and  temporally  efficient 
approach  to  the  realistic  real-time  video  tracking  problem.  By  capital¬ 
izing  on  successive  scene  temporal  dependence  cmd  target  texture,  the 
algorithm  estimates  target  rotation,  dilation  cuid  translation  p^u:^uneters . 
High  order  matrix  inversion  problems  are  circumvented  cuid  a  simple  CCD 
implementation  is  achieved  in  the  estimation  process  by  en^loying  a  matrix 
pseudoinverse  formulation  theorem  [30].  The  TSVIP  approach  is  potenti¬ 
ally  advcuitageous  in  complex  scenes  where  the  target/background  separa¬ 
tion  problem  is  coit^licated  or  time  consuming,  since  the  algorithm  may 
be  used  to  quickly  update  the  segmentation  window  or  (if  applied  in  paral¬ 
lel  over  a  number  of  sub-scenes)  to  classify  grov^js  of  pixels  as  members 
of  the  tcurget  or  background  sets .  Computer  simulation  using  smooth  ana¬ 
lytic  functions  to  represent  target  texture  confirms  the  validity  of  the 
CCD  implementable  version  emd  these  empirical  results  suggest  ^ul  approxi¬ 
mately  linear  input-output  function  for  the  resulting  tracking  processor. 

Some  obvious  cUid  desir^Uble  extensions  of  the  research  reported  herein 
are  as  follows : 

(1)  Since  the  TSVIP  eilgorithm  essentially  ignores  target  edge  in¬ 
formation,  further  study  should  be  undertaken  using  the  model 
of  Section  4.3.3  to  determine  if  this  information  is  estimable 
(with  a  reasonable  processor  complexity)  euid  useful  for  track¬ 
ing  purposes.  If  so,  it  may  be  possible  to  process  both  tex¬ 
ture  euid  edge  information  in  parallel  to  generate  more  reliable 
tracking  signals.  Note  that  this  might  also  permit  easing  of 
target  textural  constraints . 
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(2)  Derivative  estimation  techniques,  especially  those  which  are 
CCD-implementable  and  reasonably  noise  insensitive,  should  be 
investigated. 

(3)  Simulation  using  actual  scene  data  from  memeuvering  targets 
should  be  undertaken  to  further  confirm  the  validity  of  the 
TSVIP  approach. 

(4)  TSVIP  algorithm  sensitivity  to  CCD  implementation,  particularly 
with  respect  to  dark  current  and  transfer  efficiency  effects, 
should  be  studied. 

(5)  In  the  TSVIP  algorithm,  only  the  pixel  intensity  differences 
from  two  successive  scenes  are  utilized  to  estimate  a.  The 
possibility  of  using  more  than  two  scenes  should  be  investigated. 

(6)  In  situations  where  a  high  reliability  system  is  desired  (e.g., 
military  applications)  the  possibility  of  using  several  differ¬ 
ent  algorithms  in  parallel  together  with  estimate  weighting 
scheme  should  be  investigated. 

There  is  a  very  large  amount  of  research  that  czu\  now  be  defined 
for  tracking  based  on  image  analysis ,  but  the  most  important  step  is  to 
construct  a  simple  experimental  system  and  that  is  what  we  proposed  to  do 
during  the  next  year.  Such  a  system  should  emulate  as  closely  as  practi¬ 
cal  (or  simulate)  the  technology  to  be  used  in  actual  systems.  The  system 
will  allow  tracking  algorithm  development  and  testing  and  provide  a  meams 
to  start  tying  together  the  hiereurchy  of  decision  and  control  algorithm 
that  will  be  necessary  for  a  practical  system. 

The  development  of  the  TSVIP  tracking  algorithm  has  made  it  practi¬ 
cal  to  consider  the  use  of  a  microcon^)uter  as  eui  aid  to  CCD  implementation , 
or  even  as  competition  to  CCD  implementation.  Work  is  in  progress  to  ob¬ 
tain  experimental  results  based  on  the  implementation  configurations  dis¬ 
cussed  in  Chapter  6,  while  alternate  possibilities  are  also  being  pursued. 
The  sequential  (iterative)  solutions  being  considered  are  especially  in¬ 
teresting. 

Two  papers  have  been  written  and  another  is  being  planned.  The  pa¬ 
per,  "Algorithm  Development  for  Real-Time  Automatic  Video  Tracking  Systems" 
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will  be  given  at  COMPSAC  in  November  as  eUi  invited  paper  and  "A  Model  emd 
Tracking  Algorithm  for  a  Class  of  Video  Targets"  has  been  submitted  to  the 
IEEE  Treuisactions  on  Pattern  Analysis  and  Machine  Intelligence  for  publi¬ 
cation. 
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Appendix  1 :  Affine  Transform  Summary 
(a)  Mathematical  Background 

The  2-D  affine  trcmsform  has  been  used  previously  to  model  small 
target  function  perturbations  (e.g.,  see  Mostafavi  and  Smith  [40-41].) 
While  a  detailed  description  of  this  tremsform  may  be  found  in  Cans 
[42],  a  brief  summary  is  presented  here.  Basically,  the  2-0  affine 
transform  represents  a  linear  operation  on  the  argisnents  of  a  2-0  func¬ 
tion  as  follows: 

For  a  picture  function  of  the  form 

(A.1-2) 

the  general  affine  transformed  version  of  this  function  is  denoted  by 
p'(x)  -  p(x')  (A.1-2) 

where 


p(x);  X 


^1 

*2 


x^  ■  A^b 


(A. 1-3) 


It  should  be  noted  that  both  A  and  b  may  be  functions  of  time.  Thus, 
the  affine  transform,  with  respect  to  the  indices  of  p(x)  represents  a 
linear  transformation  of  the  plane  onto  itself. 

Expanding  (A. 1-3) t 


*2 


a 

c 


(A. 1-4) 


A  is  denoted  the  homogeneous  affine  trusform  matrix,  and  b  is  the 
translation  vector.  Two  well  kno%m  versicms  of  the  homogeneous  affine 
trauisform  are: 


A  -  COS0  -sine' 
sine  COS0 

m 


(dilation)  (A, 1-5) 


(rotation  about  the 
origin  through  an  (A. 1-6) 

angle,  0) 


Another  important  case  is  where  A  ■  1,  and 


(slaq^l*  translation)  (A.  1-7) 


cue  laportant  result,  which  is  quite  useful  for  image  tracJcing 
purposes,  is  the  transform  representation  of  a  rotation  through  an  angle 
3  about  a  point,  (h,k)  where  (h,]c)  ^  (0,0).  This  type  of  affine  trans¬ 
form  obviously  will  not  leave  the  target  centered  in  the  FOV  of  the  cam¬ 
era,  therefore  it  is  an  is^rtant  point.  Cans  [42]  has  shown  that  this 
rotation  may  be  represented  ais: 


Eq'  -hi  pos0  -sin^ 

••i  *  [sine  cos^  1*2 


(A.  1-8) 


which  may  easily  be  simplified  to: 


[a-[: 


-sine 


*11  R 

'  L 


(COS0-1)  -sine 


(cose-1) 


ine  h" 

-i)J  Lie. 


(A. 1-9) 


This  expression  represents  an  equivalent  rotation  about  the  origin 
(first  tern)  and  treuislat  .on  (second  term).  The  translational  term  is 
obviously  of  importance  to  the  tracking  system.  Two  additional  points 
should  be  noted: 

(1)  Combined  affine  transforms,  e.g.,  a  rotation  followed  by 
translation,  then  another  rotation,  etc.,  may  ed.so  be  repre¬ 
sented  by  a  resultant  affine  transform  of  the  form  in 

(A. 1-2) •  However,  the  order  in  which  these  transforms  occur 
is  significant  since  the  result  of  a  rotation  followed  by  a 
translation  is  not  necessarily  the  same  as  if  the  transform 
ware  instead  the  translation  first,  then  the  rotation.  This 
does  not  aiffect  the  model  developed  in  Section  4,3.2,  or  the. 
ultimate  processor,  however,  since  the  system  will  only  need 
to  estimate  the  resultant  rotational  and  translationeJ.  param¬ 
eters  (A  and  b) ,  not  the  individual  rotations. 

(2)  Slnoe  the  affine  transform  operates  on  the  target  function 
arguments  by  essentially  changing  the  coordinate  axes,  A  and 


A- 3 

b  actually  affect  the  target  fimction  in  the  inverse  way  as 
the  coordinates,  e.g.  if 

b  =  0 

we  would  actually  see  the  picture  function  magnified  by  a  fac¬ 
tor  of  Vo. 9  in  both  directions.  Likewise,  if  A  represents 
a  rotation  of  9  on  the  coordinate  system,  the  picture  function 
is  actually  rotated  by  -0.  This  fact  is  important  in  the  gen¬ 
eration  of  proper  polarity  of  control  signals. 

(b)  Affine  Transform  Examples  Using  Sampled  Scene  Data 
Using  the  computer  program  described  in  Chapter  VI  of  Appendix  5, 
numerical  examples  of  affine  perturbations  were  generated.  An  | |x| |  tar¬ 
get  was  chosen,  and  background  pixel  intensities  are  set  equal  to  zero,  for 
clarity.  Figure  30  shows  examples  of  dilation,  rotation  and  translation. 


4. SI  I.M  ».;! 

%.4t  <  «•  «  !.•< 

«.;j  *  ii  *.*4  1.M 
t.^t  <.4I  f.t«  |.4l 
%.3i  ). %  i4  %.TZ 
t.«»  t  *J  t.ri 
t.ii  t.«i  «.<«  %,n 
t.u  f.«l  t.ti  t.4t 
i.:9  f.  )t  1.44  «.ii 
%.i4  f.  n  1.M 
•  .M  i.;i  ».:• 

.M  .4;  ■#«  .4» 


I  l.*4  i.i« 

I  «.Si  It: 

I  «.•«  «.;i 

'  t.U 

1.11  I.K 
4  li  «.>4 

I  4. SI  4.:j 

».lt  4.44 
4.««  4.;t 

4.11  4.4t 
!.*•  f.S< 

.94  M 
.14  -44 


Of  4iafB 
f.«4« 


(a)  Original  Scene 


.94 

.94 

.94 

.«• 

.44 

.94 

.44 

.44 

.44 

.94 

.44 

.44 

.94 

.44 

.44 

44 

.  94 

.44 

.44 

.  44 

.44 

.44 

44 

94 

.94 

.44 

.99 

.44 

.44 

.94 

.44 

14 

.44 

.44 

.44 

.  44 

.44 

.44 

.44 

.94 

.44 

.44 

94 

.44 

.  44 

.44 

.44 

.44 

.44 

.44 

.44 

.44 

.44 

.44 

.44 

44 

44 

.94 

.44 

.44 

.44 

.41 

.44 

.14 

.  44 

.94 

.  94 

.  14 

44 

44 

.49 

.14 

.44 

.14 

.44 

.  >4 

.44 

.44 

.94 

.44 

.44 

.44 

4^ 

.99 

.  44 

.44 

.44 

.44 

.M 

.14 

.44 

.99 

.44 

.44 

44 

t.4l 

l.t) 

1.3*. 

.44 

.44 

.44 

.44 

.94 

.44 

.44 

It 

1.U 

s  w 

1.  ‘1 

.14 

.44 

.14 

.44 

.94 

.44 

.44 

44 

l.fl 

4. (4 

«.S1 

l.il 

44 

.44 

.44 

.44 

49 

44 

•4 

44 

f.<f 

$.is 

l.-l 

1. 31 

44 

44 

.99 

44 

.44 

44 

.44 

.44 

l.«4 

t.*i 

1.:; 

.44 

.44 

.99 

.44 

.44 

.14 

.44 

.  41 

.44 

.it 

.44 

.94 

.44 

.99 

.44 

.14 

.  44 

.44 

.94 

.44 

.94 

.44 

.44 

.44 

.  44 

.  94 

44 

.It 

.  It 

.44 

.94 

.44 

.44 

.  44 

.49 

.44 

.44 

.99 

44 

.44 

.44 

.44 

44 

44 

.44 

.94 

.44 

.  94 

.44 

.44 

.94 

.41 

.44 

44 

.44 

.44 

.44 

.14 

14 

.44 

.44 

.  94 

44 

.44 

44 

.44 

.44 

.44 

.44 

.44 

44 

.44 

.44 

.44 

.44 

.44 

.44 

.44 

.  94 

.44 

.94 

.44 

.44 

.  44 

44 

.94 

.44 

.44 

.44 

94 

.94 

.99 

.44 

.94 

.44 

.44 

.  M 

.94 

.44 

.44 

.44 

t.l4««  .4444  !•  .4444 

.H44  i.44«4  .4444 


Cb]  Dilation  (ci*  0. 


./#  t.jy 

.  a  4.14  4. St 

.;4  $.;4  t.JS 

.44  t.ii  1.JI 
.J9  .44  i.2i 

.44  .rt  .14 
.44  .94  .44 

.44  .*4  .44 

.44  .44  .44 

.44  .14  ,44 

.44  .44  .44 

.44  .it  .44 


44  .44 

.44  .44 

44  .44 

.44  4.;s 
t.  :i  f.«i 
f. 

%.4t  l.il  I 
4.S4  l.it  I 
t.S4  4.S7 
4.44  1.47 
%.»%  1.4! 
.44  4.4. 
.44  .44 

.44  .44 

.44 

.<4  .44 

M  .14 


.44  .44  .44  .44 


•emfftts  letM  stsn^'* 


Cc)  Rotation  (5»  45*) 
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Figure  30  Affine  Trans fora  Numerical  Examples 


Appendix  2t  •  Multidlnmsiooal  Fourier  Taansforna 
•nd  Bffacta  Doe  to  Afflna  Pertinrhations 
Th«  Fouri«r  Transfota  of  a  3-0  function  nay  b«  mitten  as 

^  rp(*;t)]  -  F(u,w)  -  /  pCxjt)e*'j^»ru’^»wJ  [i_l  ,3;cdt  (A.2-1) 

X,T  Lf  J  ” 

where  X  and  T  axe  the  regions  In  2-D  space  and  IrO  t-inwi  over  \dilch 
p(x,t)  ft  0.  Mote  that  the  integral  sign  represents  a  3-fold  integred.. 
Letting 

p'(x;t)  -  p(3c;t)  (A,2-2) 


where 


x'  •  A(t)x+b{t) 


(A.2-3) 


it  is  useful  to  consider  the  effects  of  the  affine  transfom  on 

P'(u,w)  [p(x;t)]  (A.2-4) 

especially  if  a  functional  relationship  between  p'{u,w)  and  F(u,w)  may 
be  established.  By  definition 

F'(u,w)  -  p;(x;t)e-:i2ir[uT,w]  ^^  2^, 

A  change  of  variables  is  as  follows : 


(A.2-5) 


|A(t)  °i[5i 

.tj  Lo  dltj  Lo  J 


(A.2-6) 


dx'  -  D*dx  (A.2-7) 

where  0  is  the  Jacobian  of  the  transfozaation,  i.e,, 

0  •-  |det  A(t)  I  (A.2- 

It  is  a  sinple  natter  to  show  the  inverse  transfom  of  (A.2-6},  i.e., 

r  *1  fA‘^ct)  oi  fk'i  rB{t)i 


(A.2-8) 


(A. 2-9) 


Subvtltutlaa  of  (A.2-7)  and  (A. 2-9)  info  CA,2-5)  yialds 


Making  the  substitution 

it  is  s««n  that  (A. 2-10}  may  ba  raducad  to 


P'(u,w)  •  / 


,j2iruTA-lCt)b(t) 


Idat  ACt) 


Pi  (A^ (t) )  ^luja-j2itwtdt 


{A,2-10) 


(A.2-U) 


(A.2-12) 


(A. 2-12)  may  now  ba  usad  to  show  a  variaty  of  affects  of  tha  affine 
transform  on  the  Pouriar  transform  of  tha  pictura  function.  Seme  special 
cases  are: 

(1)  A(t)  -  1,  b(t)  - 

in  which  case 
p'(u,w)  only  differs  from  P(u,w)  in  phase; 
and 

(2)  b(t}  •  q  and 

E:os9  -sinol 

iine  cosej 

in  which  case 

lA»(t)l-l  -  A(t) 

since  A  is  orthogonal.  Thtts,  wa  rotate  tha  coordinate  axis  in  Pouriar 
space  through  tha  sane  angle  as  in  the  signal  or  pictura  space. 

Equation  (A.2-12)  is* also  extremely  useful  in  that  it  easily  en¬ 
ables  calculation  of  affined-perturbed  function  spectra  when  the  unper¬ 
turbed  spectra  are  known.  Recently  Nathan  [44j  has  derived  a  special 
case  of  (A.2-12}  for  2-0  functions  ^>diere  b  ■  Q. 


bl 

b2 


i 


It  G€BTal  Ii«ayt 


ts  Estimators 


A  particularly  iaportan^  paraiMtar  dapaiadant  (linear  in  the  param¬ 
eters)  process,  corrupted  by  noise,  is  of  the  following  form: 


y  ■  0^  Q 


(A.  3-1) 


where 


y  is  the  Nxl  process  "ou^ut"  vector; 

0  is  an  MxM  known,  aeasunhle  or  calcul2kble  matrix; 
b  is  the  Mxl  parameter  vector  to  be  estimated  (denote  this  esti- 

A 

mate  by  b) ;  and 
n  is  an  Nxl  "noise"  vector. 

It  is  desired  to  estimate  b,  with  the  following  constraints  on  the 


estimate: 


(1)  to  ■ 

(2)  E{b}  - 

(3)  Cov{b} 


linearity 

unbiasedness 

"small"  or  perhaps  minimal 


(A.  3-2) 


Note  that  (1)  is  intended  to  enable  CCD  in^lementation ,  provided 
that  M  may  be  also  formed  via  CCD  type  operations. 

It  may  be  shown  (Eykhoff  [33],  Deutsch  [45])  that  the  estimate 
which  minimizes  a  loss  function  of  the  type: 

I Ir 


in  accordance  with  (1)  is  of  the  fozm: 


where 


M  -  Or 


(A. 3-3) 


(A. 3-4) 


-  (uTro)“1o^r 

i.e.,  M  equals  the  pseudoinverse  of  0,  based  on  an  R  norm,  denoted  by 

J. 

Or.  Assuming  that  0  and  q  ue  statistically  independent,  (2)  may  be 


I 


since 


A-e 


b{6}  -  E{(DTaD)“^^ROb}  +  E{(DTra)"^aTR3} 

-  b  +  E{(U'J^O)"^^R}E{g} 

if  either 

E(n)  ■  0  or 
E{(uTRn)'MR}  -  [o] 

Since  (c.f.  [33],  p.  187) 

ca7[bl  «  E{[MO^Mn-b]  [MO^Mn-b]^} 

-  (uTRD)“loTRNRU(OTRa)“^ 

where 

cov[n]  ^  N 

(3)  may  be  satisfied  by  choosing  R  ■  and  therefore 
cov{4j>  •  (n^iriu)”! 

which  can  be  shovm  [45 ]  to  be  mininnan  variance,  hence  efficient. 


(A.  3-5) 
(A.  3-6) 


(A. 3-7) 
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Appendix  4;  Cline's  Theorem  for  the  Ponnnlatioo 
of  e  Pseodolnverse  of  a  Partitioned  Matrix 


The  problem  of  forming  an  in^rse  of  a  rectangular  matrix,  A,  with 
specified  properties  has  been  studied  by  Penrose  [46],  Cline  [30]  and  a 
number  of  other  researchers.  The  pseudbinverse  of  this  MXM  real  matrix 
A,  Is  a  MXN  matrix  denoted  by  A^.  Sxas^les  of  desirable  properties  are: 


AA'^A  •  A 

a*aa'*’  «  a"*" 

(AA+)T  -  AA’’’ 


(A.4-1) 


It  is  well  known  that  if  A  has  full  column  rank,  one  inverse  of  consid¬ 
erable  interest  is  the  so-called  least  squares  inverse  denoted  by 

a"*"  -  (aTa)~1a’^  (X.4-2) 

The  properties  of  this  solution  au»  considered  extensively  In  references 
on  least  squares  estimation,  interpolation,  etc.  One  important  note  of 
particular  in^ortance  to  the  present  work  is  that  the  formulation  of 
this  pseudoinverse  requires  the  Inversion  of  an  MXM  non-singular  matrix 
(or  conversely,  the  solution  of  the  so-called  normal  equations,  which 
are  ed.so  of  order  M) .  While  nisaerous  successful  eilgorlthms  for  the  so¬ 
lution  of  this  problem  on  the  digital  computer  have  been  known  for  a 
long  time,  the  adaptation  of  these  algorithms  for  the  CCD  or  discrete 
analog  ccnputer  Is  very  difficult.  This  is  at  least  partially  due  to 
two  factors i 

(1)  In  solving  the  noanal  equations.  Gauss  Elimination  requires 
extensive  division  and  may  require  row  exchanges,  which  are 
difficult  operations  In  the  CCD  implementation^  wd 

(2)  If  the  Inverse  of  (A^A)  is  cooputed  directly,  e.g.,  via  Craun- 
ers  rule,  the  memory  requirements  emd  calculations  become  ex¬ 
cessive. 

Cline  [30 ]  studied  the  problem  of  computing  the  pseudoinverse  of 


parti tiOMd  aatricas  of  tha  fom: 
A  -  [C  ;  V] 


(A. 4-2) 


Basically  this  solution  is  as  follows >  { 

For  a  parti tionad  I*x(R+(M-R))  matrix  ^ 

] 

A  -  [0  i  V]  i 

[0  :  . 

whare 

C  -  (I-OT''’)V  (A.4-3b) 

K  -  [I  +  {i-cx)v^(a*)^n‘’’v{i-cx)  ]"^  (a.4-3c) 

Although  at  first  it  may  sacm  that  tha  cooplexity  of  tha  ahova  aquations 
savaraly  limit  thair  utility,  two  casnants  ara  in  ordari 

(1)  ‘Ria  formulation  of  V**  and  X  raqnira  matrix  invarsions, 
ho««avar  tha  ordar  of  thesa  invarsicns  is  significantly  ra- 
ducad,  a.g.,  K  is  only  an  (M-R)  x  (M-R)  matrix;  and 

(2)  In  Saction  5. 4. 4. 3  it  is  show  that  if  0  and  V  hava  cartain 
advantagaoxis  propartias  (which  tha  matricas  and  G  in  fact, 
do  possass)  that  considarable  reduction  in  tha  conylaxity  of 
the  preceding  aquations  may  result. 


0 

c 


b'''-o'^vc*-o*v  (i-c''‘c)  X7^  (o*)  ^a'*’  (i-vc*)' 
c*+  (i-c*c)  xy"^  (u'*’)  '^0*  (i-vc*) 


(A.4-3a) 


A- 12 


t 


t 


© 


USE  GRAMIAN  TO 


DO  ' - 


5.  Subroutine  DERIV(IOP) 


START  DERIV(IOP) 


DO 

I-l/NUlT/AL 


IOP=3 


IOP-2 


VI 

a 

IND(I,1)+1 

V2 

a 

IND(I,2) 

V3 

a 

VI -1 

V4 

a 

V3 

VS 

- 

V2+1 

V6 

» 

V2 

V7 

a 

D0 

VI 

a 

IND{I,1)+1 

V2 

= 

mD(I,2) 

V3 

= 

Vl-2 

V4 

- 

Vl-1 

VS 

a 

V2+1 

V6 

» 

V2-1 

V7 

- 

2.O*D0 

DERR(I,1)  -  (P(V1,V2)-P(V3,V2))/V7 
DERR(1,2)  =  (P(V4,V5)-P(V4,V6))/V7 


X  »  D0*(IND(I,1)-11 
Y  -  D0*{IND{1,2)-11 


TC  VT7C 


Dl-3.0 


Dl*(SIN(Y/2.5) )/(Y/2.5)+0.3*Y+2.0 


D2=0 


D2»(COS (X) )/X-{SIN(X) )/(X*X) 


DERR(I,1)  *  D1*D2 


DPI- (COS {Y/2 . 5 ) ) / (Y/2 . 5 ) 
DP2-(SIN(Y/2.5))/((Y/2 , 5 ) *Y/2 . 5 ) ) 


D4- (DP1-DP2 ) *0 . 4+0 . 3 


'•'^^X-0?  ^  >  D3-2.0 


DERR(I,2)  -  D3*D4 
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Appendix  5.2  Program  TSVPO  Listing 
WOGRA.M  TSVPD 

C  SOURCftFILK  IS  TSVDS:  RELOCATE  =?LE  IS  TSVDR 
C  If'AGE  TRACKING  SPnJLATIOM  PROGRAM 
C  THIS  version  estimates  trams,  a  homo.  AFFINE  PAR. 

C  INCLUDES  GESTALT  SEGMENTATION  OPTION 

C  WRITTEN  RY  H.  J.  SCVALKOFF,  UMIV.  OF  VA. 

C  UPDATED  3/14/70?  NO  DEBUG  STATE'^ENTS 

C  *★*★*★★**★★*»***★»  **★***★*★**■*★★**★★★★•***★★★★+★★★★*★ 

DIMBISION  F(21  ,21  )  ,LAM(21  ,21  ),RACV:(21  ,21  ) 

DIMENSION  P(21 ,21 ) ,P1  (21 ,21 ) ,A(2,2) .5 (2 ) , DIFF (400) , IND (400 , 2) 
dimension  DERR(40fi,2)  ,REST(2) 

DIMENSION  PC(4r'0,2)  ,PCINV(2,4rin)  ,aC(2) 

COMMON  A,B,D«^,NUMCAL,F,F1  ,LAM,LA*M  ,P,P1  .DIFF,IND 
CO ‘.'ON  NU  MV  A  L  ,  DE  WR  ,  JM I N ,  JMA  X ,  KM  I N  ,  KM  A  X ,  B  ES  T 
CO MMON  ONORM , PC, PC I MV , C I NV , AC 
REAL  LAM,LAM1 

3  CALL  CHRS (27,1 2,1 2B) 

i.RITEd  ,2) 

2  for maT( "SELECT  OUTPUT  DEVICE  OPTION"  ,/,«  1 -CRT  DISPLAY",/, 

A "6- TELETYPE") 

PPADd  ,★)  IDIS 
■.■.■PITE(  IDIS,30) 

30  FOR.MAT ("»*★★*♦★★*★★  TAYLOR  SERIES  VIDEO  IMAGE  PROCESSOR  ★★★★ 

) 

d  IJRITE(IDIS,15) 

15  F0Rf'AT(3(/), "INPUT  OPTION-"  ,/,"  1 -COMT.  ?  2-STOP,  3-OLD  PIC.") 

PFAD( IDIS,*)JO 
GO  TO  (35,1000,799)30 
C  DO  IS  THE  SPATIAL  SAMPLING  INTERVAL 

35  ;;RITE(  IDI5,40) 

40  FORMATC  INPUT  D^,THE  SAMPLING  INTERVAL") 

READ(  IDIS,*)D(" 

C  INITIAL  TARGET  FUNCTION  *★★★★★* 

r."JMCAL»0 
CALL  TARG 

C  * •*★■*■* *■* » »F0 H M  LA'*'0A,  THE  CHARACTERISTIC  FN.  ★★***★★★★★■»★ 

•.;RITE(  id  is,  75) 

75  .PORMATC  INPUT  T.ARGET  HEIGHT  AND  WIDTH  PARAMETERS") 

HEAD(  Ii)IS,»)MEIGT,l*.TDTH 
C  WEIGT  A  .•IIDT-t  ARE  MULTIPLES  OF  DO 
CALL  LA.’<D(HEIGT, WIDTH) 

c  GE'IFRATE  (CONSTANT)  BACKGROUND  FN.  *★*★★★★ 

C  BACKGROUND  HAS  NORMAL  (VARIABLE)  STATISTICS 
WRITE (ID IS. 350) 

350  FORM.ATC'IMPUT  BACKGROUND  MEAN,  STD.  DEV,") 

READ!  iniS,*)B»»,PDEV- 
DO  400  J»l ,21 
DO  400  K*l ,21 
CALL  GaUSS(BDEV,RH,T1  ) 

BACK(J,K)»T1 

400  continue 
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C  ★•»★■*★★★★*★  add  RAMDH'T  (GAUSSIAN)  ‘•’OISli  SCENE  ★★★x**** 
C  NOTE  THAT  NOISE  IS  ALWAYS  >=0  IN  MAGNITUDE 
WRITE! I DIS, 505) 

60b  FORMAT!  "IN  PUT  RANE30M  SCENE  NOISE  MEAN,  STD.  DEV.'*) 
HEAD!IDIS.*)SNM,SNDEV 
C  GENERATE  COMPLETE  INITIAL  SCENE 

DO  500  Jal ,21 
DO  500  K*I  ,21 

402  CALL  GAUSS !SnDEV,SNM,T0) 

Ir!T0.LT.0.0)GO  TO  402 

?!  J,:<)aF!J,lC)  xLAM!J,K)  +!  1  .0-LAM!J,!C  ) )  ★BACK!  J,!C) 

P!  J,!C)»P!J,K)+T0 
500  CONTI  HUE 

C  *★•***■*•»**  PROVIDE  SCENE  DISPLAY  OPTION 

WRITE! IDIS.510) 

3  10  FORMAT!"  PRINT  SCENE  DATA?") 

READ!  IDIS,-*')  J1 
IF!J1 .EO.0)GO  TO  799 
CALL  CHPS!27,I2.12B) 

WRITE!  IDIS,530) !  !P!J,!<:)  ,?C=I  .2! )  ,J=»1  ,21 ) 

530  FORMAT!  10!/)  ,2  I  !/ ,  2  HF4. 2 ,  1  X  ) ) ) 

WR  I TE !  I  DI S ,  905 )  D0 ,  HE  I GT.  WI DTH ,  BM ,  BDEV 
..RITE!I0IS,532) 

532  FORMAT!  5!/)  ,4'^X,"SCP'1E  DATA*  INITIAL  SCFNE") 

C  irrrir*ilr*itirir^iriirkitir  FORM  PETURBED  SCENE 

C  INPUT  AFFINE  DISTORTION  PARAMETERS 

799  WRITE!  ID  IS,  BOO) 

£300  FORMAT!  I  0!/) ,"  INPUT  ALPHA  A  THETA  (DEGREES)  ") 
READdDIS,*)  ALPEX,THAEX 
..RITE!  IDIS,3I0) 

310  FORMAT! "MOW  THE  TRANSLATION  VECTOR, B") 

READ! IDIS,*)B! 1 ) ,R!2) 

C  FORM  HOMOGEiNEOUS  AFFINE  MATRIX 

A! I ,1 )»AL?EX*C0S!THAEX/57.2958) 

A!2,2)«A! 1,1) 

Ad  ,2)*-l  .'^★ALPEX*SIN!THAEX/57.295R) 

A!2,l  )»-l  .0  <A!  1 ,2) 

iJU’fCAL»l 

CALL  TARG 

CALL  LAMD 

DO  880  J-1,2I 

DO  880  K- 1 , 2 1 

812  CALL  GAUSS!SNDEV,SNM,TT3) 

IF!T0.LT.0.0)GO  TO  812 

PI  !J,'<:)«F!J,K)*LA''(J,1C)  +(1  .0-LAM!J,K))*BACK!J,K) 

PI  !J,.<)«PI  !J,!0  +T0 
880  CONTINUE 

C  PETUR«ED  SCENE  DISPLAY  OPTION 

•■•.RITE!  IDIS, 521  ) 

321  FOpVATC'DiSf’LAY  PERTURBED  SCPIF?") 

READ!  IDIS,*)J2 

IF! J2.EO.0)GO  TO  450 
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CALL  CHRS(27, 12, 128) 

'..RITHC  1015,53^)  ((PI  (J,K)  ,K»1  ,21  )  ,J=I  ,21  ) 
l-.'RnH(  IDIS,522) 

b22  FORMaT(5(/)  ''PERTURBED  SCET,'E  DISPLAY") 

WRITE(IDIS,92CT)A(1 , 1) .A( 1 ,2) ,B( 1 ) ,A(2. 1 ) ,A(2,2 ) ,B( 2) 
.VRITE(IDIS,2P'P») 

200  F0R?'AT( 3 (/), "ACCEPTABLE  PERTURBATION?") 

READ(IDIS,*)J5 
IF(J5.E0.'^)Gn  ID  799 
C  HERE  FILTER  SCENE(S)  IF  DESIRED 

C  LOOK  AT  SCENE  DIFFERENCES 

C  USE  PI  TO  HOLD  DIFFERENCE  SCENE 

450  DO  3'^0  l3l,2I 

DO  300  J=I ,21 
PI (I,J)=PI (I,J)-P(I,J) 

300  CONTINUE 

i-iRrTE(  IDIS,302) 

302  FnRMAT(5(/) , "DISPLAY  SCENE  DIFFERENCES?") 

READ(  IDIS,^)  IDIFF 

IF( IDIFF.EO.0)GO  TO  309 

CALL  CHRS(27, 12, 123) 

l.RITE(  rDI5,530)  ( ( P 1  ( I ,  J)  ,  J=  1 ,2 1  )  ,  1  =  1  ,  21  ) 

».'RITE(IDIS,3(^3) 

30S  F0RMAT(5(/)  ,40X, "DIFFERENCE  BETVl.  ORIG.  A  PERT.  SCENES") 

309  ;;rite(IDIS,3I0) 

310  F0RMaT(5(/)  ."AUTOMATIC  GESTALT  SEGMENTATION  OPTION?") 

REAC(  IDIS,*)  IGES 

IF( IGES.EQ. 1 )G0  TO  360 

C  *★★*★*★***  WIMDOIV  FUNCTION  FORMULATION 

550  WRITF( IDIS, 570) 

57v0  FOR?:^AT("IMPUT  WINDOW  FN.  LI  Ml TS-JMIN ,  J>'AX ,KMIM  ,KNA.X"  , /, 

A "RECALL  JMIN,KMIN>=2»  UMAX ,KMAX<=20" ) 

R E AD  ( I D I S  ,  I?l ,  J?UX ,  KM  P' ,  K  MA  X 

C  IrTr-Jrir-xieit -kifirkirir  SA?fPLING  STRATEGY  OPTIONS 

C  RECALL  MAX.  SCENE  DIM.  IS  (N-l)X(N-l) 

C  SAMPS  IS  SKIP  SAMPLED 

620  ..RITF(  IDIS. 622) 

622  FOR  “ATC  INPUT  SKIPHORIZ.  JSKIPVERP') 

RFADdDIS,*)  ISKPH.ISKPV 
CALL  SAMPS(  ISKPH.ISKPV) 

C  *■* •*★★!» ★★■**★*★*★★  FORM  D.  THE  DIFFERENCE  VECTOR  *** ♦-»»■*■****'* 

67b  CALL  DVFCT 

C  ★★•*★.★**★★★  FORM  DERRIVATIVE  ESTIMATES  ****★**★★★* 

WRITH( IDIS,690) 

690  FOR  MAT  ("IN  PUT  DERRIVATIVE  ESTIMATION  OPTION") 

READdDIS,*)  I0PT2 
C 

CALL  DERIVdOPT2) 

GO  TO  355 

C  ■**•★*★•*★•*•  A U TO .  GESTALT  SEGMENTATION  OPTION 

360  ;VRrTFdDrS,362) 

362  FORMATC'INPUT  WIND  U.L.H.  CORNER  ROUN  DS  .  X  .Y  ,D  IM.  ADFP  OPT  ") 
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REACC I DIS , x) ICSX  »  rCSY , ICFX , ICFY . IWy , IWY. 1^072 
JMI.MalCSX 
j''AX=j'jaM+rv^x 
fCMIMalCSY 
KMAX-fCfUM+IW 
941  CALL  SA?<;P!5(.! ,  I  ) 

CALL  DVECT 
CALL  DERIV(I0PT2) 

GO  m  94fJ 
392  XMrN=K?^AX 

•<’'<AX=<?nM+IVIY 
IFCKMIN.LT. ICFY)GO  TO  941 
;<MrN=icsY 
K^:AX=fC’a:i+iwY 
J?.!IM=JHAX 
J?^AX=JMIN+rwX 
IFCJMIM.LT. ICFX)GO  TO  941 
GO  TO  8 

C  ★■X'*'****'**  THRESHOLD  IMG  OPTION 

355  ITHRaO 

C  ***x*************  TAYLOR  SERIES  ^  L.S.  PARAMETER  VECTOR  EST. 

C  OiiLY  PERMITS  TRANSLATION,  ROTATION  A  'fAGNIFICATION 

C  FORM  PC 

940  DO  045  I  =  1,NU,WAL 

X=D'^’^(  IND(  1, 1  )-l  1 ) 

Y=D0x(rMD(I,2)-in 

PC (1,1 )=X*DFRR(I,1 )+Y*DERR(I,2) 

PC(I,2)=X*OERP(I,2)-Y*DERR(I, 1 ) 

945  CONTINUE 

C  FORM  GRAMIAjM  TO  SEE  IF  DC  V^AS  FULL  COLU’^J  RANK 

U1I=0. 

DI2«0. 

DI3=0. 

DI4=0. 

022=0. 

D23=0. 

024=0, 

033=0. 

044=0 , 

DO  947  I  =  1,NUWAL  • 

D11«PC(I,1)*PC(I,1)4-01  1 
UI2«PC(  I,1)*PC(I.2)+01  2 
D1  3»PC(  1 , 1  )  ★DERR  ( 1 , 1  )  +01 3 
01  4=cc  (1,1)  ★LEHR  ( 1 , 2)  +D1  4 
022=PC(I,2)*?C(I,2) -*022 
D23=PC(  I,2)*nERR(I  ,1  )  +023 
D24-PC(  1,2)  xOEHRd  ,2)  4024 
PSSaOERR ( I , I ) +OFRR ( I , 1 ) +033 
:}34aOE«R(  1 ,  1  IxOFRRC  1,2) +034 
D/'.4  =  0E!M'(  I  ,2)  +DERR(  1,2)  +044 
947  CONTINUE 
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C  NO’.'/  FORM  Dc-n^fli'INANT 
Li!  “  D33*D44.-D34*D34 
02»DI  3*024-Dl  4*023 
D3»0i  3*044-0!  4*034 
04*023  *044-024*034 
Db»D23*034-024*033 
Drt»D!  3*034-0!  4*033 
P!  =01  I  *022*01  -023*04+024*05) 

P2»D!  2*(D!  2*0!  -023*03 -K) 24 *06) 

P3=D1 3*(0I 2*04-022*03+024*02) 

P4=Di  4*(0i  2*05-022*06+023*02) 

D=P!  -P2+P3-P4 
V.’RITEC  1013,915^)0 

9n  F0R''AT(3{/)  ."GRAMIAN  DETERMINANT  =  '•,E1!^.2) 

C  FORM  PCJMV 

CALL  PSEUD 
C  FOR’f  PCINV*C 
V!  !=<^. 

V!2=0. 

V2!»0. 

V22=<^. 

DO  950  K=!.NU'-n/AL 
VI  1=FCIMV(  1  ,fC)  *OERR('<,  1  )+V!  1 
V!  2  =  PCrNV(  1  .K)  *nEP.P(K,2)+V!  2 
V2!=PCIMV(2,K)*DESSC:<,1  ) +V21 
V22=PCINV(2,K)*DERR('<,2)  +V22 
950  continue 
C  FORM  PCINV*D 

XI  =0. 

X2=0. 

CO  970  !<=!  .NU’.n/AL 
X!  =  PCINV(!  ,K)*DIFF(X)+X! 
X2=PCINV(2,K)*DIFF(«C)  +X2 
970  CONTINUE 

C  FOR;/  -PC*PCINV*G;  RESULT  IN  PC 

DO  955  !C=!,NUMVAL 
FC<X,!  )  =  PC(K,  1  )*V!  !+0C(’<.2)*V2I 
PC(K.2)  =  PC(K,!  )*V!2+?C(X,2)*V22 
955  CONTINUE 

C  FORM  C 

DO  960  I»1,NUMVAL 

PC(I,1  )=DERR(1 .1  )-PC(IJ  ) 
PCCl,2)»QERR(I.2)-PCa.2) 

960  CONTINUE 

C  FORM  REST  VECTOR 

CALL  PSEUD 

C  RETURNS  CINV  IN  PCINV 

BFSTC 1 )=0. 

5EST(2)=0. 

DO  063  I=I,NUMVAL 

REST(  1  )  =  PCINV(  1,1)  *niFF(  I )  +RFST(  1  ) 

.FE3T(2)  =  ?CIMV(2, 1)  *niFF(  I)+'^FST(2) 
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963  CONTINUE 
C  Fn«?»  AC 

ACC  1  )=X1-Yt  I  *REST<  I  i -V I  2*BESTC  2) 

AC(2)=X2-V2I  *BESTC  1  ) -V  22*RF5TC  2) 

C  FORM  ESHMATJES  OF  ALPHA  &  THETA 

ACC  I  )aAC(  1  )  ■•■1  .0 
■IHETA»ACC2)/ACCI  ) 

AL?MA=ACC  ]  )*C  1  .  f? +C  THETA  ★THETA  )/2.0) 
rriET.'‘=THETA*57.3C^ 

C  CONPUTF  R  VECTOR  FST.  ERROR  RASED  ON  L2  VECTOR  NORV 
E1*FC  1  )-RESTC  1) 

E2=*5C2)-BESTC2) 

ERRORaSORTCEI ★EI+E2*E2) 

C  ♦•.v*'**'*'*'»r*'*r'*'**'**OI SPLAY  OF  ALC/OR  ITHM  RESULTS  ★★•******Tk-*r 

IFC  roFS.EO. I )G0  TO  370 

979  IFC  ITriR.EO.  1  )G0  TO  980 
THRESH»0,0 
iSAVE^rru’cvAL 

980  Call  C’-'RSC27, 12, 128) 

WRITE C IDIS,900) 

900  FOPVATCSC/)  - FOR  THIS  CASE  THE  FOLLOWING  WAS  P'FUT - '' ) 

’.'•P  I TH  ( I D I S  ,  905  )  DO .  HF I GT ,  IVI  DT^ ,  B  N ,  R  DEV 
905  FORMaTC  5C/)  ,''D0",3X,"“EIOWT".4X,  “WIDTH"  ,5X  ,  “BMEAM"  ,  5X  ,  “P  CEV-"  ,/ 
CF 

A6.3,4X)) 

WRITECIDIS,9I 4)AL?Ey,THAEX 

914  F0R''ATC3C/)  .“ALPHA*  “,3X,r8.5,“THETA=  ",3X,Fa.5) 
;;ritecidis.9i5) 

915  r'^RMATCSC/)  .“AFFINE  TRAilSrORV  PARA’'ETERS  ===*====“  ) 

MrflTEC IDIS,920)AC I  , I ) ,A  C 1 ,2) ,BC I ) .AC2. 1 ) ,AC2,2) ,RC2) 

920  format C 2 C/) , 1^X,“A=“.3X,2CF6.4,2X) , “B*“ ,F6. 4 ,/ , i 5X ,2 CF6 . 4 , 2X) , 
^  * 

f  r 

A6.4) 

WHI  TE  (  I  DIS  .925  )  JMIN  .JHA.X  .XNIM  ,XMAX 

925  FORMATC3C/)  ,"V;iTH  WINDOW  LIMITS - "  "  JNIN="  ,  1 2 .5X . "  Jf.^A  X="  .  I  2 

A“KNIM=".  I2.5X,“KMAX»",I2) 
..'l<ITE(IDIS,926)rSAyp,rSfCm,ISKPV,ISAVE 

926  F0RMATC3C/) , "SAMPLING  STRAT.  " , 3X ,  "WOR IZ.  INCR.  "  .3X  ,  "VERT.  I'  :R 

.3 

AX, "RAW  NO.  SAMPLES",/,5X.ri ,I7X,I2,I2X,I2,14X,I3) 

<iRITE(  ID  IS,  92  7)  10912,  iriPT 

927  F0RMATC3C/)  ,"DEHR.  EST.  Op-TIOM"  .5X  ,"L.S.  WEIGHTING  OPTION",/, 9 
II 

A  22X  12) 

’.’iRITEC  I  DIS,  02^)  ITHR, THRESH,  ICOUNT 

928  F0RMATC3C/)  ."THRESHOLD  OPTION",  5X,"T>'RE5WOLD"  ,5X,"Anj.  NO.  SA'' 
ES 

A",/.9X,  II  ,l6:<,r4.2.15X,r3) 

..tITEClDrS,930)RC  1  )  .RESTC I  )  ,RC2)  .RESTC2) 

930  E0R;.!AT(5C/)  ,5X,"R  C  EXACT)  ="  ,F6 . 4 ,  |  px  .  "R  C  EST )  ="  ,  FA.  4  , /, 


-w. 
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?.I5X.F6.4,  I9X,F6.4) 

.'.hITSC  II}rS,932)  4C(  1  )  .AC(2) 

932  rnw,’'AT(5(/)  ,5X,"AC(  1  )  ="  ,  3X ,  F6.  4 , 1  £^X  , AC(  2 )  =  ,  F6 . 4) 

•.•.■RrTE(  IDrS,933)ALPt-iA.  WETA 

933  FnR?lAT(2(/)  ."ALPHA  (£571=  "  .F3.5  ,5X .  "THETA  (EST)»  ",F8.5) 

■/.■RrTE(IDrS,93i  )FRROR 

931  rOR>'AT(3(/)  ,"B  VECTOR  ERROR. L2  NORM.=  ",F15.12) 

GO  10  8 

C  ★★★★**'<r**GEFTALT  SEG.  OPTION  DISPLAY ★*★•*★+* 

3 70  iVH  I  TC  <  I  D 1 3 . 92 5 )  J V I M  ,  J?U X  , X.M  IN  .  1C??A X 

mRITc  (1015.930)8  Cl  )  .OFSTC  I  )  .  B  ( 2 )  .  nESTC2 ) 

;-iR  I TE  ( I D I S ,  933 )  ALPH  A .  THETA 
GO  TO  392 
1  END 

C 

C  SUBROUTINES  ★★★Tt*-*******?**-*'**** 

SUBROUTINE  TARG 

DIHPISION  F(21  ,21  )  ,LAM(21  ,21  )  ,3ACK(21  ,21  ) 

DIMENSION  P(21  ,21  )  ,P1  (21  ,21  )  ,A(2,2)  ,B  (2)  ,  DIFF (400)  ,IND(4O0,2) 
DIMENSION  OERR(40^,2) ,3EST(2) 

DIMENSION  ?C(40O,2) , PCINV(2,400) ,AC(2) 

COMMON  A,B.Do,MUMCAL,F.Fl  ,  LA '',LAMI  .  p  ,  a  I  .DIFF.IND 
COMMON  irj  Jn/AL ,  DERR ,  J’'I N ,  J MA X  , KM P! , KMAX  .BEST 
.CO’MGM  OMORM.PC,  PCINV .CPIV ,  AC 
REAL  LAM, LAM 1 

C  THIS  ROUTINE  GE'IERATES  THE  INITIAL  AND  PERTURBED  TARGET  FNS 
DO  200  J=l ,21 
DO  200  !<=1 ,21 

C  THE  POINT  ( J,K)=( 1 1 , 1 1 )  CORRESP.  TO  (X,Y)=(O,0) 

X=D0*(J-1 I ) 

Y=Do*(;<-n ) 

IFCNUMCAL.EO. 1)120,125 
120  XP=X*A(  1  , 1  )+Y*A( !  ,2)+B(  1  ) 

Y?=X*A(2, I )+Y*A(2,2)+B(2) 

X=XP 

Y=YP 

125  IF(X.EO.O.O)GO  to  154 
AI  =  (SIN(X))/X+1  .0 
GO  TD  155 

154  A1=2.0 

155  IF(Y.EO.0.0)GO  TO  158 
A2»(SIN(Y/2.5 ))/(Y/2.5)  +2.0+0.3*Y 
GO  TO  159 

158  A2=3.0 

159  FUM1=A1*A2 
185  F(J,K)=FUN1 
200  CONTINUE 

RETURN 

END 

SUBROUTINE  LAND (H, VI) 

DIMENSION  F(21  ,21  )  , LAM  (21  ,21  )  ,BAC'^(21  .21  ) 

DIMENSION  P(2I  ,21  ),  =  I  (21  ,21  ),A(2,2)  ,B(2),DIFF(400)  ,IMD(40'^,2) 
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Di;/ENSrr)M  DRRR  .BERK?) 

nrMF>!Sin-i  PC{^P^,2)  ,  .20i'iy(2.4^Pi)  ,  AC(2) 

COMMOM  A,3,D^.NU’‘CAL,F,F1  ,LAM,LAM1  .P.Pl  ,DIFF,rND 
CO .’/.WN  MU’.fV aL , DFRR  ,  JM  IN ,  JMAX  ,KM IM . K?.'AX , BEST 
Cn MMOM  QMnR M , PC , PC I NV , C I NV , AC 
REAL  LAM, LAM I 

C  FORMS  THE  IMITIAL  AND  PERTURBED  CHAR.  FUNCHONS 
C  OBJECT  IMITIAL  BOUNDARY  IS  FN.  OF  HFIGT  &  WIDTH 
DO  300  J=l  ,21 
DO  300  K=1 ,21 
X=Do*<J-n) 

Y=0n*(IC-1 1  ) 

IF(NUMCAL.EO. I )320,325  .  . . 

320  XP=X*A(  I  ,  1  )+Y*A(l  ,2)+B(l  ) 

YP«X*A(2,I  )+Y*A(2,2)+9(2). 

X=XP 
Y=YP 

C  LATER  MAfCE  CHARACTERISTIC  FUNCTION  USER  ADJUSTABLE 

325  IF(ABS(X).GT.(H*D0))GO  TO  265 

IF(ABS{Y)  .GT.  (W*DO))G0  TO  265 
T1  =  I 

GO  TO  285 
265  T1=0.0 

285  LAM<J,iC)=Tl 
300  CONTINUE 
RETURN 
END 

SUBROUTINE  SAMPS  ( ISK’PHS ,  ISKPVS ) 

DrOSlON  F(2I  ,21  )  ,LA^f(21  ,21  )  ,BACK(21  ,21  ) 

DI  -TENSION  P(  21  ,21)  ,  PI  (21  ,21)  ,  A  (2,2)  ,B  ( 2 )  ,  DIFF ( 400) ,  IND(40<^,2) 
DIMENSION  DERR (400, 2) , BEST (2) 

DIMENSION  PC(400,2) ,PCINV(2,410) .aC(2) 

COMMON  A,B,D0,NUMCAL,F,F1  ,LA,M,LA^n  ,P,P1  ,DIFF,INO 
CO  MMON  MU  MVAL ,  DERR ,  J’^IN ,  JMAX  , <M  IN ,  KMAX ,  BEST 
COMMON  ONORM,PC, PCINV,CIMV,AC 
REAL  LAM, LAM  1 

C  RECTANGULAR  SKIP-SAMPLED  STRATEGY 

C  NUMVAL  IS  MO.  OF  VALID  ROV.-S  IN  IND(  ,  ) 

NUMVAL=0 

K1=I 

J=JMIN 

K=KMIN 

20  IND(K1,1)=J 

IND(K1 ,2)=fC 
NUMVAL=NUMVAL+l 
K1=K1+1 
K=K+I5KPHS 

IF(K.GT.KMAX)G0  to  105 
GO  TO  20 
105  J=J+ISKPVS 

TF(  J.GT.  JMAX)GO  TO  106 
K=KMIN 


:si3  PAOi  IS  srsT  vjalitt 
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GO  TO  20 
106  RHTUHM 
END 

SUBROUTINE  DVHCT 

DIMENSION  F(21 ,21 ) ,LAM(21 , 2 1 ) , BACK (2 1 ,21 ) 

DII^RISION  P(21  ,21), PI  (21  ,21  ) ,  A  (  2, 2)  ,B  (  2)  ,DIFF  ( 400) ,  IND  ( 400,2 ) 
DIMR4SI0N  DERR (400, 2) ,BE3T(2) 

DIMENSION  PC(400,2) ,PCINV(2,400) ,AC(2) 

COMMON  A,B,DO,NUMCAL,F,Fl ,LAM,LAMI , P, PI ,DIFF, IND 

CO  Mf'ON  MUMV  AL ,  DERR  ,  J’'  IN ,  UMAX ,  KM  I N ,  K  MA  X ,  PEST 

CO.'O.'ION  QMORM,PC,PCIMV,CINV,AC  ... 

REAL  LAM,  LA  Ml 

C  FORMS  D  VECTOR,  BASED  UPON  SAMPLING  STRATEGY 
CO  J=*I,NUMVAL 

DIFF( J)=P1 ( IND(J,1 ) ,IMD(J,2)) 

100  CONTINUE 
RETURN 
END 

SUBROUTI.NE  DERIV(IOP) 

DIMENSION  F(21 ,21 ) ,LAM{21 ,21 ) ,BACK(21 ,21 ) 

DIME^iSION  P(21  ,21  )  ,  PI  (21  ,21  )  ,  A (2, 2 )  , B  (2  )  ,  DI FF ( 400) ,  IND (400, 2) 
DI.’OSION  DERR  (400,2)  ,0EST(2) 

DIMEIiSIO.I  PC(400,2)  ,PCriV(2,400)  ,AC(2) 

COMMON  A,P.,D0,MUMCAL.F,F1  ,LAV,LA>'1  ,P,!31  ,DIFF,IND 
CO  M'AON  MU  MV  AL  ,DERR ,  I N ,  UMAX ,  KM  IN ,  KMAX ,  BEST 
CO  MMON  ONOR M , PC , PC I N V , C I N V , AC 
REAL  LAM, LAM 1 

C  FORMS  SPATIAL  DERRIVATIVE  ESTIMATES  "^ROM  P{  ,  ) 

C  OPTS  DETERMINES  DERRIVATIVE  ESTIMATION  ALGORITHM 

C  I0P=1  ;  USES  I  SI  +  1,I0P=2?  USES  I-l  AI+I 
C  I0P=3;  COMPUTES  EXACT  DERIVATIVES 

DO  750  I  =  l,NUf‘VAL 
GO  m  ( 100,300,400) lOP 
100  vi  =  niD(  I,  I ) +1 
V2=IND( 1,2) 

V3=VI  -1 

V4=V3 

V5=V2+1 

V6=V2 

V7=:D0 

GO  TO  500 

300  V1=IND(  1,1  )-»-l  ... 

V2»IND( 1,2) 

V3=Vl -2 

V4=V1-1 

V5-V2+1 

V6=V2-1 

V7=2.0»D0 

GO  TO  500 

C  EXACT  DERIVATIVE  COMPUTATION 

400  X=D0*(IND(r, I )-I I ) 


I-ACX 15  Bfsr  9«u.m  nucn«Ab;. 

^  Tt)  uDc 

Y=Dror(  rMO(  i,2)-i  I ) 

C  FORM  DF/DX  A. 

IF(Y.EO.O.O)GO  TO  1(? 

DI  =  (SIN(Y/2.5  ))/(Y/2.5)  +0.3*Y+2.O 
GO  m  12 
10  01=3.0 

12  IF(X.EO.O.O)GO  TO  20  _  .  . 

D2=(C0S(X))/X-(SIN(X))/(X*X)  . 

GO  TO  22 

20  02=0.0 

22  DERH(I,1 )=01*D2 

C  FORM  DF/OY  1 

IF(Y.EO.0.0)GO  TO  30 
D?l=(COS(Y/2.5))/(Y/2.5) 

DP2=(Sni(Y/2.5))/((Y/2.5)*(Y/2.5)) 

D4=0.4*(DP1-DP2)-K5.3 
GO  TO  32 
30  04=0.3 

32  IF(X.EO.0.0)GO  TO  40 

03=  (SIN(X))/X+1  .0 
GO  TO  42 
40  03=2.0 

42  DERRCI ,2)=D3*04 

GO  TO  750 

50'^  CEfiPC  r,  I  )  =  (P(V1.,V2)-P(V3,V2))/V7 

DERR(I,2)=(P(V4,V5)-P(V4,V6))/V7 
750  CONTINUE 

RETURN 
END 

SUBROUTINE  PSEUD 

C  COMPUTES  THE  PSEUDO  INVERSE  OF  AN  NSIZEX2  MATT?  IX 

DIMENSION  F(2I ,21 ) ,LAM(21 ,21 ),3ACK(21 .21 ) 

DIMENSION  P(21  ,21  )  ,  PI  (21  ,21  )  ,A(2,2)  ,  B  (2 )  ,  DI FF  ( 400  )  ,  IND(  400, 2) 
DIMENSION  DERR(40O,2) ,BEST(2) 

OIMEISION  PC(400,2; ,PCINV(2,400) ,AC(2) 

COMMON  A,3,D0,NUMCAL,F,F1  .LAM.LAMI  .P,P1 .DIFF.IND 
CO  MMOM  MU  MV  AL , DERR , J  ■  f  I N , J  MA  X , KM I N , KM A  X , B  EST 
COMMON  QNORM,PC,PCINV,CINV, AC 
REAL  LAM.LAMI 
T1»0. 

T2»0. 

T1  2-0. 

DO  100  T»1 .NUMVAL 
■n=PC(I,l)*PC(I,l)+Tl 
T2»PC(I,2)*PC(I,2)  +T2 
T12=PC(I,1)*PC(I,2)+T12 
10O  CONTINUE 

DET«TI  ★T2-TI2*T12 
QNORM=DET 

IFCDET.EO.O.  )G0  TO  200 
OWECP=l .0/DET 
DO  200  J=l  ,NU?(VAL 
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2P0 


PCIMVd  ,J)< 
PCINV(2.J)> 
CONTINUE 


i(T2*PC(J 
(Tl  *PC(J 


RETURN 

END 

BiO$ 


•  I  )-Tl  2*PC(J,2))*DRECP 
,2)-Tl2*PC(J ,1 ))*DRECP 


Appendix  6  -  Simple  Physical  Interpretation  of 
the  TSVIP  Approach  and  Some  Numerical  Examples 
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(a)  Physical  Interpretation  in  1-D 

Some  insight  into  the  euialysis  of  Section  5.4  may  be  obtained  by  con¬ 
sidering  a  simple  1-D  example.  Consider  a  1-D  function  f(x;t),  which  is 
subjected  to  a  time-varying  translation,  i.e., 

f(x;t2)  =  f[x  +  6(t2-t3^)  ;ti]  (A. 6-1) 

A  first  order  Taylor  series  approximation  to  this  function  would  be : 

f(x;t2)  :  f(x;ti)  i-  6(t2-ti)  (A. 6-2) 


which  may  be  rewritten  as ; 


'  f  [xjtp]  -  f  [x;ti  ] 


(A. 6-3) 


3x 


Therefore,  for  any  value  of  Xq  in  the  sensor  remge,  f(xQ;t)  at  times  tj^  emd 
t2  and  an  estimate  of  Q*  ^ 

(A. 6-3)  . 


dx 


may  be  used  to  estimate  6(t2-t2)  using 


A  physical  interpretation  of  this  approach  is  shown  in  Figure  31.  Two 
successive  samples  (i.e.  "scenes")  of  a  1-D  function  f(x;t)  are  shown.  For 
simplicity,  in  the  time  interval  laetween  tj^  euid  t2)f(x/t)  is  translated  by 
an  amount,  5  ,  as  shown  in  Figure  31.  Due  to  this  translation,  the  functional 
intensity  at  point  Xq  and  time  ti,  i.e.  fCxQ.-tj^),  shown  as  in  the  figure 
is  no  longer  the  same  at  this  same  point  at  time  t2 ,  i  .e . ,  at  P4.  It  is 
clear,  however,  that  at  time  t2 ,  the  value  of  the  function  would  be  equal 
to  the  value  at  P5 ,  i.e.  f(xj3;t2)  =  f(xQ;t2^).  (Also  note  that  this  treuis- 
lation  would  be  accomplished,  with  the  x  axis  positive  direction  as  shown, 
by  making  f(x;t2)  =  f (x  -5 ft^)  since  the  function  at  t=t2  is  actually  a  spa¬ 
tially  delayed  version  of  f(x;tx).)  The  sensor  "line  of  sight"  is  chosen 
such  that  the  function  intensity  is  sampled  at  point  Xq  in  both  scenes. 


/ 


3f(x  ;t  )  f(x,  :t  )  -  f(x  :t 


la  +1  1 


3x 


>1 


/ 

V 

v/N.  Sensor  "Line  of  Sight",  i.e.. 
Sensor  Coordinate  X 

0 


Figure  31  Simple  Physical  Interpretation  of  TSVTP  Approach  in  1-D 
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Thus,  to  employ  the  TSVIP  approach,  we  proceed  as  follows: 

(1)  At  time  t  =  ti  and  x  =  Xq,  the  spatial  derivative  of  f(x;ti), 

i.e.  ,  '  is  estimated  using  the  value  of  the  function 

3x 

at  points  and  x^.^  as 

3f  (Xr.;ti )  .  f(x+l;t)  -  f(x-i;t)  (A. 6-7 

3x  '  ^+l”’^-l 

(2)  Using  the  values  of  the  function  at  x©  and  times  t^  emd  t2 
(i.e.,  these  intensities  shown  ais  Pi  euid  P4  in  Figure  20)  and 
the  above  derivative  estimate,  6 (t2-ti)  is  calculated  using 
(A. 6-3) . 

It  is  also  interesting  to  note  if  =  0,  the  algorithm 

oX 

obviously  fails.  This  is  due  to  the  fact  that  f(x;ti)  has 
no  1-D  "texture,"  or  equivalently,  the  "matri^'  Dc  (actually 
a  1x1  matrix  in  1-D)  in  (5.4.4.2-10)  is  "singular."  This  is 
analogous  to  the  2-D  case  examined  in  Sections  5. 4. 4. 2  and 
5. 4. 4. 3. 

(b)  1-D  Numerical  Examples 

(Example  1) 

In  this  simple  example ,  the  functions 


f  (x ; ti )  =  COS  X  and 

f  (x;t2)  =  cos  (x-hS  ) 

are  considered.  For 

A  =  11/6  = 

.52360  (six  times  the  Nyquist  rate) 

and 

5  »  -2Tr/20  “  -.15708 

(30%  of  A) 

,  three  consecutive  scunple  function 

values 

would  be : 

X 

cos  X 

cos  x+6 

.52360 

.36652 

.86603 

.93358 

1.04720 

.89012 

.50000 

.62932 

1.5708 

1.4137 

.00000 

.15645 

For  simplicty,  Xq  is  chosen  as  the  second  sample,  i.e.  Xq=1. 04720.  The 
calculations,  as  outlined  in  part  (a)  would  proceed  as  follows; 
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(1) 


at  t  *  tj^ 

Sf  (Xr.;ti)  .  0  -  .86603  ^ 

3x  '  1.5708-. 52360 

3f 

(Note:  the  exact  — (xQ;tj^) 


-.8270 

*  -sin  (1.04720) 


-.866) 


(2)  Using  (A. 6-3)  and  the  above  data. 


«(t2-ti)  = 


.62932-. 50000 
-.8270 


=  -.1564 


%  error 


I-.1564  -(-.15708) 
+.15708 


=  0.43% 


(Example  2) 

Using  the  same  data  as  in  Example  1,  but  with  a  larger  translation, 
i.e.,  let  5(t2-ti)  =■- 2  ip'lO  =  -.62832  (120%  of  A  ) 

at  Xq;  6(t2-ti)+x  =  1.047720  -  .62832  =  .41888 
cos(Xq+6)  =  cos  (.41888)  =  .91355 
Therefore,  from  (A. 6-3  ),  the  translational  estimate  is 


<S(t2-ti) 


.91355-. 50000 
-.8270 


-.5006 


%  error 


|-.5006-(-. 62832)  I 
+.62832 


20.3% 


Clearly,  the  effect  of  the  assumption  as  to  a  "small"  perturbation  on  the 
algorithm  estimates  is  evidenced  in  the  above  two  examples. 


(c)  2-D  Numerical  Examples  Using  the  TSVIP  Algorithm 

A  simple  numerical  example  of  the  TSVIP  algorithm  procedures  is  given 
here.  The  raw  data  used  for  this  example  is  taken  from  the  sample  scene 
described  in  Chapter  VI.  The  initial,  perturbed  and  difference  scenes  are 
shown  in  Figure  32.  For  simplicity,  N  =  4  was  chosen,  and  the  four  points 
resulting  from  a  raster  scam  are  shown  circled  in  the  figure .  Derivatives 
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Figure  32  Initial,  Perturbed  and  Difference  Scenea  with  Four  Points 
Used  for  the  Numerical  Example 
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were  estimated  using  (A. 6-4) .  It  should  also  be  noted  that  the  scene  pixel 
location  (6,6)  corresponds  to  the  (arbitreurily  chosen)  coordinate  system 
origin,  with  positive  Xj^  and  X2  axes  downward  and  to  the  right  respectively. 
In  this  example,  only  translation  was  considered,  and  the  following  exact 
parameters  were  used: 

«  =  1.0  (no  dilation)  (A.6-5a) 

0  =  0.0  (no  rotation)  (A.6-5b) 

'.ll 


b  = 


.1 


(translation  of  0.5 A 
up  eind  to  the  left) 


(A.6-5C) 


Using  these  points  from  Figure  32  ,  the  following  sample  data  would  re¬ 


sult: 


d  = 


.03160 

.0726 

.0762 

.0368 


(A. 6-6) 


G  = 


Pc  = 


-0.2025 

0.47675 

0.2025 

0.57675 

0.19450 

0.61925 

jO. 19450 

0.61925 

‘'0.07485 

0.15585" 

0.07485 

-0.15585 

-0.16275 

-0.08495 

-0.16275 

0.08495 

(A. 6-7) 


(A. 6-8) 


Using  the  above  data  and  (5.4.4.3-31)  yields 

-.06718  0 


Pc^Pc  = 


.06301 


(A. 6-9) 


and  from  (5.4.4.3-29) 
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Vj^  =  4.05x10 


-3 


(A. 6-10) 


Recall  (5. 4. 4. 3-8),  i.e.. 


C  =  (I-PcPc 


(A. 6-11) 


C"*"  = 


4.4.3-13),  (A 

6-8)  and 

(A. 6-9)  we  get 

' .03535 

.71113^ 

-.03535 

.71113 

C  = 

.06485 

.32705 

j. 06485 

.32705_ 

.3-12)  , 

3.2398 

■3.2398 

5.9437 

-5.9437  1 

_  .58035 

.58035 

.26691  .2669lJ 

(A. 6-12) 


(A. 6-13) 


So  the  estimated  translational  vector,  b,  is  obtained  using  (A. 6-13) ,  (A. 6 
and  (5.4.4.3-28),  i.e., 

b  =  C^d  (A. 6-14) 

or,  numerically, 

'  .10135 


b  = 


(A. 6-15) 


.09063J 

It  should  be  noted  that  this  estimate  is  in  excellent  agreement  with 


-6) 


the  exact  parameter  values  given  in  (A.6-5c) . 


(a)  Sensor 


(c)  Fixed  Weight 


(b)  Surnner 


(d)  Variable  Tap  Weight 
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